Nov 03 16:13:34.957122 ip-10-0-143-64 systemd[1]: Starting Kubernetes Kubelet... Nov 03 16:13:35.409318 ip-10-0-143-64 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:13:35.409318 ip-10-0-143-64 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 16:13:35.409318 ip-10-0-143-64 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:13:35.409318 ip-10-0-143-64 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:13:35.412765 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412634 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:35.412765 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412765 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:35.412765 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412769 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412773 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412777 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412781 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412783 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412786 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412789 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412805 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412808 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412811 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412813 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412816 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412819 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412821 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412824 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412826 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412829 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412832 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412835 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:35.412879 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412844 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412847 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412850 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412852 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412855 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412857 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412860 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412862 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412865 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412868 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412871 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412874 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412876 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412879 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412883 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412886 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412888 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412891 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412894 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:35.413323 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412896 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412899 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412901 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412906 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412910 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412913 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412916 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412920 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412922 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412926 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412928 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412931 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412934 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412937 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412939 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412942 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412944 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412948 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412951 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:35.413773 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412953 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412956 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412958 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412961 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412964 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412967 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412970 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412972 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412976 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412979 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412982 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412986 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412990 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412993 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.412997 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413000 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413002 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413005 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413008 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:35.414234 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413011 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413014 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413016 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413019 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413022 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413024 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413026 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413029 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413032 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413034 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413037 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413047 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413049 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413052 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413054 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413057 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413059 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413062 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413065 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413068 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:35.414684 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413071 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413074 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413076 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413079 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413082 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413084 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413087 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413091 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413942 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413950 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413954 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413957 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413959 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413963 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413966 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413969 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413972 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413974 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413977 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413980 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:35.415212 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413982 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413985 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413987 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413990 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413993 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413996 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.413999 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414002 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414004 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414007 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414009 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414014 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414017 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414022 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414024 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414028 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414030 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414033 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414036 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414038 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:35.415678 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414041 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414043 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414046 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414048 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414051 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414053 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414056 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414059 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414061 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414064 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414067 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414069 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414072 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414075 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414077 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414080 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414083 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414085 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414087 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:35.416187 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414090 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414092 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414095 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414098 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414100 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414102 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414106 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414108 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414111 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414113 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414116 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414118 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414121 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414123 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414126 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414129 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414132 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414134 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414136 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:35.416647 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414139 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414142 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414144 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414146 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414149 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414151 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414157 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414160 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414162 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414165 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414167 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414170 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414173 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414175 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414178 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414180 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414183 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414185 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414187 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414192 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:35.417129 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414194 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414197 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414199 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414202 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414205 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414210 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414213 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414216 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414219 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414222 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414224 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414227 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414230 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414233 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414235 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414238 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414753 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414758 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414762 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:35.417618 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414764 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414767 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414769 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414772 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414774 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414777 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414779 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414782 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414784 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414787 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414789 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414807 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414812 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414814 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414817 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414819 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414822 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414824 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414827 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414829 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:35.418115 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414832 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414834 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414837 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414839 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414842 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414844 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414848 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414851 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414854 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414856 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414859 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414861 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414864 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414867 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414869 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414872 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414874 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414877 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414879 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414882 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:35.418594 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414884 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414887 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414889 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414892 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414897 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414899 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414902 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414904 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414907 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414909 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414912 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414914 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414917 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414919 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414922 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414924 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414927 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414929 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414932 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414934 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:35.419136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414937 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414940 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414943 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414946 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414948 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414951 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414953 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414956 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414959 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414961 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414964 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414966 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414969 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414971 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414974 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414976 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414982 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414986 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414989 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:35.419627 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414992 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.414996 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415000 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415003 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415005 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415008 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415010 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415013 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415015 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415018 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415020 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415023 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415026 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415029 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415033 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415035 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415038 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415041 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415043 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:35.420130 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415046 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415048 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415050 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415053 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.415055 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415904 2577 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415917 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415926 2577 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415932 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415937 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415941 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415946 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415950 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415954 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415957 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415960 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415964 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415967 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415970 2577 flags.go:64] FLAG: --cgroup-root="" Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415973 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415976 2577 flags.go:64] FLAG: --client-ca-file="" Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415979 2577 flags.go:64] FLAG: --cloud-provider="external" Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415981 2577 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 16:13:35.420575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415986 2577 flags.go:64] FLAG: --cluster-domain="" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415991 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415994 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.415997 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416001 2577 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416006 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416009 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416012 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416016 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416019 2577 flags.go:64] FLAG: --contention-profiling="false" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416022 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416025 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416028 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416031 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416036 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416039 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416042 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416045 2577 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416055 2577 flags.go:64] FLAG: --enable-server="true" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416061 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416067 2577 flags.go:64] FLAG: --event-burst="100" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416070 2577 flags.go:64] FLAG: --event-qps="50" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416073 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416076 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 16:13:35.421190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416079 2577 flags.go:64] FLAG: --eviction-hard="" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416083 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416087 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416090 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416093 2577 flags.go:64] FLAG: --eviction-soft="" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416096 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416099 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416102 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416105 2577 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416109 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416112 2577 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416115 2577 flags.go:64] FLAG: --feature-gates="" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416120 2577 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416123 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416129 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416132 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416136 2577 flags.go:64] FLAG: --healthz-port="10248" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416138 2577 flags.go:64] FLAG: --help="false" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416142 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-143-64.ec2.internal" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416145 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416148 2577 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416151 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416154 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 16:13:35.421820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416158 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416161 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416164 2577 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416167 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416170 2577 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416173 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416177 2577 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416180 2577 flags.go:64] FLAG: --kube-reserved="" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416183 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416186 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416189 2577 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416192 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416195 2577 flags.go:64] FLAG: --lock-file="" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416198 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416201 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416204 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416211 2577 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416214 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416219 2577 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416222 2577 flags.go:64] FLAG: --logging-format="text" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416225 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416229 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416231 2577 flags.go:64] FLAG: --manifest-url="" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416234 2577 flags.go:64] FLAG: --manifest-url-header="" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416240 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 16:13:35.422400 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416243 2577 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416253 2577 flags.go:64] FLAG: --max-pods="110" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416256 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416260 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416263 2577 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416265 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416269 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416272 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416275 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416284 2577 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416287 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416290 2577 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416294 2577 flags.go:64] FLAG: --pod-cidr="" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416296 2577 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416299 2577 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416302 2577 flags.go:64] FLAG: --pods-per-core="0" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416305 2577 flags.go:64] FLAG: --port="10250" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416308 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416311 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ed36073cce201fb9" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416314 2577 flags.go:64] FLAG: --qos-reserved="" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416317 2577 flags.go:64] FLAG: --read-only-port="10255" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416321 2577 flags.go:64] FLAG: --register-node="true" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416323 2577 flags.go:64] FLAG: --register-with-taints="" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416327 2577 flags.go:64] FLAG: --registry-burst="10" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416330 2577 flags.go:64] FLAG: --registry-qps="5" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416335 2577 flags.go:64] FLAG: --reserved-cpus="" Nov 03 16:13:35.423060 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416338 2577 flags.go:64] FLAG: --reserved-memory="" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416341 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416346 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416349 2577 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416352 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416355 2577 flags.go:64] FLAG: --runonce="false" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416358 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416362 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416365 2577 flags.go:64] FLAG: --seccomp-default="false" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416368 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416371 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416374 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416377 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416381 2577 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416384 2577 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416386 2577 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416391 2577 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416394 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416397 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416399 2577 flags.go:64] FLAG: --system-cgroups="" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416402 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416408 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416410 2577 flags.go:64] FLAG: --tls-cert-file="" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416413 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416418 2577 flags.go:64] FLAG: --tls-min-version="" Nov 03 16:13:35.423723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416421 2577 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416423 2577 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416426 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416429 2577 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416432 2577 flags.go:64] FLAG: --v="2" Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416437 2577 flags.go:64] FLAG: --version="false" Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416443 2577 flags.go:64] FLAG: --vmodule="" Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416447 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416451 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416596 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416600 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416603 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416606 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416609 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416613 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416616 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416619 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416621 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416624 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416626 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416629 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:35.424376 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416631 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416633 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416637 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416640 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416643 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416645 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416647 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416650 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416671 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416675 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416678 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416681 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416683 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416686 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416689 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416691 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416694 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416697 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416700 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416702 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:35.424893 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416705 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416708 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416710 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416713 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416716 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416718 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416721 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416724 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416727 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416729 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416732 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416735 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416737 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416740 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416744 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416747 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416749 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416753 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416756 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416759 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:35.425382 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416762 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416764 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416767 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416770 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416772 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416775 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416778 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416781 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416783 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416787 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416789 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416806 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416809 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416812 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416814 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416817 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416819 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416822 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416825 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:35.425881 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416829 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416831 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416834 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416836 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416839 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416841 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416844 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416848 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416851 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416853 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416856 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416858 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416861 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416863 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416866 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416868 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416871 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416873 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416876 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:35.426318 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416878 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:35.426776 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416881 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:35.426776 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416883 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:35.426776 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416887 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:35.426776 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416890 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:35.426776 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416892 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:35.426776 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416894 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:35.426776 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416897 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:35.426776 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416900 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:35.426776 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416903 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:35.426776 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416905 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:35.426776 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416907 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:35.426776 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416911 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:35.426776 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416915 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:35.426776 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416919 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:35.426776 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416922 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:35.426776 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.416950 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416992 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416997 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.416999 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417003 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417006 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417008 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417011 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417014 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417016 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417019 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417021 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417024 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417026 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417029 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417031 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417034 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417036 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417039 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:35.427246 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417041 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417046 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417049 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417052 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417054 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417057 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417060 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417062 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417065 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417068 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417071 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417073 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417076 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417079 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417083 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417087 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417090 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417095 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417098 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:35.427966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417100 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417103 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417106 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417108 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417111 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417113 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417116 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417119 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417121 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417124 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417126 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417129 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417132 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417134 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417138 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417141 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417143 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417146 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417150 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417152 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:35.428436 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417155 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417158 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417160 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417162 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417165 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417167 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417174 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417177 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417180 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417182 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417186 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417189 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417191 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417194 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417196 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417199 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417201 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417204 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417207 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:35.428966 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417209 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417211 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417214 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417217 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417219 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417222 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417225 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417228 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417230 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417235 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417238 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417241 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417244 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417246 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417249 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417251 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417254 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417256 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417259 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417261 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:35.429428 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417264 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417267 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417269 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417274 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417276 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417279 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417281 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417284 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417286 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.417288 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.424624 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.424647 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.424667 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.424673 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424730 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424735 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424739 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424742 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424745 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424748 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:35.429970 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424751 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424754 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424756 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424759 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424761 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424764 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424766 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424769 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424772 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424774 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424776 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424779 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424782 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424784 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424787 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424790 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424806 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424809 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424811 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424814 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:35.430456 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424816 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424819 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424822 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424825 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424828 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424830 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424833 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424835 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424839 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424844 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424846 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424849 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424851 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424854 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424857 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424859 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424862 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424865 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424867 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424869 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:35.431052 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424872 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424875 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424878 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424880 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424882 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424885 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424887 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424890 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424893 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424896 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424899 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424901 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424904 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424907 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424910 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424913 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424918 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424921 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424923 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:35.431544 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424926 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424928 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424931 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424933 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424936 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424938 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424941 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424943 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424946 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424948 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424951 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424953 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424956 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424958 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424961 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424964 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424966 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424969 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424971 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:35.432133 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424973 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424978 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424982 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424985 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424988 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424991 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424993 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424996 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.424998 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425001 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425004 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425007 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425009 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425012 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425015 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425017 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425020 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425022 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425025 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:35.432625 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425028 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:35.433097 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425030 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:35.433097 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425033 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:35.433097 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.425065 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:13:35.433097 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425212 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:35.433097 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425216 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:35.433097 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425218 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:35.433097 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425221 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:35.433097 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425223 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:35.433097 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425226 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:35.433097 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425228 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:35.433097 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425232 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:35.433097 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425236 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:35.433097 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425239 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:35.433097 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425241 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:35.433097 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425245 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425247 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425249 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425252 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425254 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425257 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425259 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425262 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425265 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425267 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425270 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425273 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425275 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425278 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425280 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425283 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425285 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425287 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425289 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425292 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:35.433448 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425294 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425297 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425299 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425302 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425304 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425307 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425309 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425312 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425314 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425317 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425319 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425322 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425325 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425327 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425330 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425332 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425335 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425337 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425339 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425342 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:35.433952 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425344 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425347 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425349 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425353 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425355 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425358 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425361 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425363 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425366 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425369 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425371 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425374 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425376 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425379 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425381 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425384 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425386 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425389 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425392 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425394 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:35.434505 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425397 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425399 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425403 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425405 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425408 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425411 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425413 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425416 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425418 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425420 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425423 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425426 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425428 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425430 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425433 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425435 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425438 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425441 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425443 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425446 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:35.435006 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425448 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:35.435488 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425450 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:35.435488 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425453 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:35.435488 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425455 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:35.435488 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425458 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:35.435488 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425460 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:35.435488 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425462 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:35.435488 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425465 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:35.435488 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425467 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:35.435488 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425469 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:35.435488 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425472 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:35.435488 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425475 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:35.435488 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425478 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:35.435488 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425481 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:35.435488 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425484 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:35.435488 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.425508 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425539 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425543 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425545 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425548 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425551 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425553 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425555 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425558 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425561 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425563 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425565 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425568 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425570 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425573 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425576 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425579 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425581 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425584 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425586 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:13:35.435870 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425588 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425591 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425594 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425596 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425599 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425602 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425604 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425607 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425609 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425611 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425614 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425616 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425619 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425622 2577 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425624 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425627 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425632 2577 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425635 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425637 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:13:35.436413 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425640 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425642 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425645 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425647 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425650 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425652 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425655 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425657 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425660 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425663 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425665 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425668 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425670 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425673 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425675 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425677 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425680 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425682 2577 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425685 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425687 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:13:35.437368 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425690 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425692 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425694 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425697 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425699 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425702 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425704 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425707 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425710 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425714 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425718 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425721 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425724 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425727 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425729 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425732 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425734 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425737 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425739 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:13:35.437934 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425742 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425745 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425749 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425752 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425755 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425758 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425760 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425763 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425765 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425767 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425770 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425772 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425775 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425777 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425780 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425782 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425785 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425789 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425803 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:13:35.438566 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425805 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:13:35.439136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425808 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:13:35.439136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425811 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:13:35.439136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425814 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:13:35.439136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425817 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:13:35.439136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425819 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:13:35.439136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425822 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:13:35.439136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425825 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:13:35.439136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425828 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:13:35.439136 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:35.425830 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:13:35.439136 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.425895 2577 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 16:13:35.439136 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.430535 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 16:13:35.439136 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.431511 2577 server.go:1014] "Starting client certificate rotation" Nov 03 16:13:35.439136 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.431620 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 16:13:35.439136 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.431671 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 16:13:35.439136 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.434446 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:13:35.439136 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.435677 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:13:35.447076 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.447052 2577 log.go:25] "Validated CRI v1 runtime API" Nov 03 16:13:35.454008 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.453987 2577 log.go:25] "Validated CRI v1 image API" Nov 03 16:13:35.454911 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.454883 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 16:13:35.458554 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.458520 2577 fs.go:135] Filesystem UUIDs: map[2ce85333-6709-41c1-8952-c530e2aa739b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ecd882ce-6f3b-4a6d-9895-209189573d62:/dev/nvme0n1p4] Nov 03 16:13:35.458634 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.458553 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 16:13:35.464656 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.464520 2577 manager.go:217] Machine: {Timestamp:2026-11-03 16:13:35.463261016 +0000 UTC m=+0.390067537 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099605 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25b77bc0e7f44a77e44724648fcbe4 SystemUUID:ec25b77b-c0e7-f44a-77e4-4724648fcbe4 BootID:64cfe853-8223-4424-ba46-656247125e46 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f0:7a:c5:67 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f0:7a:c5:67 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:3d:2b:0b:cc:b1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 16:13:35.464734 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.464657 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 16:13:35.464833 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.464821 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 16:13:35.467385 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.467356 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 16:13:35.467532 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.467387 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-64.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 16:13:35.467576 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.467543 2577 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 16:13:35.467576 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.467552 2577 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 16:13:35.467576 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.467567 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:13:35.468352 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.468340 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:13:35.469097 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.469085 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 16:13:35.472844 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.472826 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 16:13:35.472996 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.472985 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 16:13:35.473654 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.473645 2577 kubelet.go:493] "Attempting to sync node with API server" Nov 03 16:13:35.473689 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.473659 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 16:13:35.473689 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.473669 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 16:13:35.473689 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.473679 2577 kubelet.go:405] "Adding apiserver pod source" Nov 03 16:13:35.473689 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.473688 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 16:13:35.474137 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.474114 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:13:35.477286 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.477253 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 16:13:35.482711 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.482688 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:13:35.483390 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.483377 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 16:13:35.483442 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.483419 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 16:13:35.483589 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.483579 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 16:13:35.483617 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.483596 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 16:13:35.483617 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.483603 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 16:13:35.483617 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.483608 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 16:13:35.483617 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.483614 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 16:13:35.483717 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.483620 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 16:13:35.484207 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.484198 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 16:13:35.484245 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.484209 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 16:13:35.484245 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.484216 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 16:13:35.484245 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.484222 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 16:13:35.484245 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.484232 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 16:13:35.484245 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.484240 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 16:13:35.486920 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.486908 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 16:13:35.486920 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.486921 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 16:13:35.491141 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.491125 2577 server.go:1263] "Started kubelet" Nov 03 16:13:35.491373 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.491302 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 16:13:35.491480 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.491367 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 16:13:35.491480 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.491400 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 16:13:35.492063 ip-10-0-143-64 systemd[1]: Started Kubernetes Kubelet. Nov 03 16:13:35.492698 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.492528 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 16:13:35.493697 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.493678 2577 server.go:317] "Adding debug handlers to kubelet server" Nov 03 16:13:35.496900 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.496880 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 16:13:35.497688 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.497668 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 16:13:35.498645 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.498629 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 16:13:35.498761 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.498741 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-64.ec2.internal\" not found" Nov 03 16:13:35.499269 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.499250 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 16:13:35.499269 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.499247 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 16:13:35.499395 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.499278 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 16:13:35.499395 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.499382 2577 reconstruct.go:97] "Volume reconstruction finished" Nov 03 16:13:35.499395 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.499390 2577 reconciler.go:29] "Reconciler: start to sync state" Nov 03 16:13:35.499630 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.499591 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 16:13:35.499630 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.499627 2577 factory.go:55] Registering systemd factory Nov 03 16:13:35.499630 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.499634 2577 factory.go:223] Registration of the systemd container factory successfully Nov 03 16:13:35.499927 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.499877 2577 factory.go:153] Registering CRI-O factory Nov 03 16:13:35.499927 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.499892 2577 factory.go:223] Registration of the crio container factory successfully Nov 03 16:13:35.499927 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.499914 2577 factory.go:103] Registering Raw factory Nov 03 16:13:35.499927 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.499929 2577 manager.go:1199] Started watching for new ooms in manager Nov 03 16:13:35.500936 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.500920 2577 manager.go:319] Starting recovery of all containers Nov 03 16:13:35.510602 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.510187 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 16:13:35.511577 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.511556 2577 manager.go:324] Recovery completed Nov 03 16:13:35.519030 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.519012 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:13:35.522374 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.522353 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:13:35.522477 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.522393 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:13:35.522477 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.522407 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:13:35.523059 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.523045 2577 cpu_manager.go:226] "Starting" policy="none" Nov 03 16:13:35.523128 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.523058 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 16:13:35.523128 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.523081 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 16:13:35.523128 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.523049 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:13:35.523848 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.523828 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:13:35.523948 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.523862 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:13:35.523948 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.523873 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:13:35.525367 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.525354 2577 policy_none.go:50] "Start" Nov 03 16:13:35.525409 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.525374 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 16:13:35.525409 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.525384 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 16:13:35.527485 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.527469 2577 policy_none.go:44] "Start" Nov 03 16:13:35.535494 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.535466 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-64.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 16:13:35.535605 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.535473 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-64.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 16:13:35.536273 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.535464 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-64.ec2.internal.18e4950889007955 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-64.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-64.ec2.internal,},FirstTimestamp:2026-11-03 16:13:35.491086677 +0000 UTC m=+0.417893204,LastTimestamp:2026-11-03 16:13:35.491086677 +0000 UTC m=+0.417893204,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-64.ec2.internal,}" Nov 03 16:13:35.544931 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.544903 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 16:13:35.555150 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.554979 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-64.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 16:13:35.555261 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.554979 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 16:13:35.555261 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.554979 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 16:13:35.555261 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.554931 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-64.ec2.internal.18e495088adde0d3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-64.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-64.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-64.ec2.internal,},FirstTimestamp:2026-11-03 16:13:35.522373843 +0000 UTC m=+0.449180366,LastTimestamp:2026-11-03 16:13:35.522373843 +0000 UTC m=+0.449180366,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-64.ec2.internal,}" Nov 03 16:13:35.586078 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.567561 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 16:13:35.586078 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.570278 2577 manager.go:341] "Starting Device Plugin manager" Nov 03 16:13:35.586078 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.570319 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 16:13:35.586078 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.570332 2577 server.go:87] "Starting device plugin registration server" Nov 03 16:13:35.586078 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.570640 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 16:13:35.586078 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.570652 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 16:13:35.586078 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.570824 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 16:13:35.586078 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.570945 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 16:13:35.586078 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.570954 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 16:13:35.586078 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.571381 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 16:13:35.586078 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.571422 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-64.ec2.internal\" not found" Nov 03 16:13:35.586078 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.579290 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 16:13:35.586078 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.579313 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 16:13:35.586078 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.579334 2577 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 16:13:35.586078 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.579367 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 16:13:35.586078 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.579663 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-78rh8" Nov 03 16:13:35.586078 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.584782 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-64.ec2.internal.18e495088ade4470 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-64.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-64.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-64.ec2.internal,},FirstTimestamp:2026-11-03 16:13:35.522399344 +0000 UTC m=+0.449205869,LastTimestamp:2026-11-03 16:13:35.522399344 +0000 UTC m=+0.449205869,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-64.ec2.internal,}" Nov 03 16:13:35.592642 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.592620 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-78rh8" Nov 03 16:13:35.609697 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.609600 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-64.ec2.internal.18e495088ade7a83 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-64.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-64.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-64.ec2.internal,},FirstTimestamp:2026-11-03 16:13:35.522413187 +0000 UTC m=+0.449219714,LastTimestamp:2026-11-03 16:13:35.522413187 +0000 UTC m=+0.449219714,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-64.ec2.internal,}" Nov 03 16:13:35.641719 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.641685 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:13:35.671579 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.671505 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:13:35.672618 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.672597 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:13:35.672727 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.672633 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:13:35.672727 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.672645 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:13:35.672727 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.672681 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-64.ec2.internal" Nov 03 16:13:35.679782 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.679756 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal"] Nov 03 16:13:35.679859 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.679849 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:13:35.680812 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.680771 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:13:35.680939 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.680824 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:13:35.680939 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.680839 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:13:35.683112 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.683094 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:13:35.683258 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.683244 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:13:35.683303 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.683281 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:13:35.683943 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.683926 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:13:35.684047 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.683954 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:13:35.684047 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.683966 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:13:35.684047 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.683932 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:13:35.684047 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.684030 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:13:35.684047 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.684043 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:13:35.686228 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.686213 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal" Nov 03 16:13:35.686308 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.686238 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:13:35.687055 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.687034 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:13:35.687136 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.687064 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:13:35.687136 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.687077 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:13:35.695236 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.695219 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-64.ec2.internal" Nov 03 16:13:35.695331 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.695242 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-64.ec2.internal\": node \"ip-10-0-143-64.ec2.internal\" not found" Nov 03 16:13:35.717856 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.717827 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-64.ec2.internal\" not found" node="ip-10-0-143-64.ec2.internal" Nov 03 16:13:35.720870 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.720852 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-64.ec2.internal\" not found" node="ip-10-0-143-64.ec2.internal" Nov 03 16:13:35.735044 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.735017 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-64.ec2.internal\" not found" Nov 03 16:13:35.800573 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.800533 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10d52772a6531c8bd8223292030e42c2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal\" (UID: \"10d52772a6531c8bd8223292030e42c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:13:35.800573 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.800571 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10d52772a6531c8bd8223292030e42c2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal\" (UID: \"10d52772a6531c8bd8223292030e42c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:13:35.800820 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.800590 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9d137c53e9ec4edb6f2bb3a754f3bb53-config\") pod \"kube-apiserver-proxy-ip-10-0-143-64.ec2.internal\" (UID: \"9d137c53e9ec4edb6f2bb3a754f3bb53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal" Nov 03 16:13:35.835134 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.835094 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-64.ec2.internal\" not found" Nov 03 16:13:35.901125 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.901096 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10d52772a6531c8bd8223292030e42c2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal\" (UID: \"10d52772a6531c8bd8223292030e42c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:13:35.901125 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.901126 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9d137c53e9ec4edb6f2bb3a754f3bb53-config\") pod \"kube-apiserver-proxy-ip-10-0-143-64.ec2.internal\" (UID: \"9d137c53e9ec4edb6f2bb3a754f3bb53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal" Nov 03 16:13:35.901309 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.901142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10d52772a6531c8bd8223292030e42c2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal\" (UID: \"10d52772a6531c8bd8223292030e42c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:13:35.901309 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.901169 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10d52772a6531c8bd8223292030e42c2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal\" (UID: \"10d52772a6531c8bd8223292030e42c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:13:35.901309 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.901202 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10d52772a6531c8bd8223292030e42c2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal\" (UID: \"10d52772a6531c8bd8223292030e42c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:13:35.901309 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:35.901204 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9d137c53e9ec4edb6f2bb3a754f3bb53-config\") pod \"kube-apiserver-proxy-ip-10-0-143-64.ec2.internal\" (UID: \"9d137c53e9ec4edb6f2bb3a754f3bb53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal" Nov 03 16:13:35.936280 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:35.936195 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-64.ec2.internal\" not found" Nov 03 16:13:36.021579 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:36.021539 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:13:36.025301 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:36.025276 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal" Nov 03 16:13:36.036978 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:36.036959 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-64.ec2.internal\" not found" Nov 03 16:13:36.137657 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:36.137613 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-64.ec2.internal\" not found" Nov 03 16:13:36.238154 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:36.238071 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-64.ec2.internal\" not found" Nov 03 16:13:36.338649 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:36.338612 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-64.ec2.internal\" not found" Nov 03 16:13:36.399336 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:36.399304 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:13:36.431595 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:36.431566 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 16:13:36.432180 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:36.431748 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 16:13:36.432180 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:36.431742 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 16:13:36.438864 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:36.438834 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-64.ec2.internal\" not found" Nov 03 16:13:36.539535 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:36.539502 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-64.ec2.internal\" not found" Nov 03 16:13:36.576437 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:36.576412 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:13:36.577672 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:36.577651 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:13:36.577836 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:36.577688 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:13:36.577836 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:36.577701 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:13:36.582975 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:36.582937 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d137c53e9ec4edb6f2bb3a754f3bb53.slice/crio-cd11594e79cefb0bb847b42f8639cfaf6757665400c707c490511b666e78a220 WatchSource:0}: Error finding container cd11594e79cefb0bb847b42f8639cfaf6757665400c707c490511b666e78a220: Status 404 returned error can't find the container with id cd11594e79cefb0bb847b42f8639cfaf6757665400c707c490511b666e78a220 Nov 03 16:13:36.583580 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:36.583552 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d52772a6531c8bd8223292030e42c2.slice/crio-2c6d7e35341f617447534ea331414166e75c5c6220a636690c5e2db938cc4ad7 WatchSource:0}: Error finding container 2c6d7e35341f617447534ea331414166e75c5c6220a636690c5e2db938cc4ad7: Status 404 returned error can't find the container with id 2c6d7e35341f617447534ea331414166e75c5c6220a636690c5e2db938cc4ad7 Nov 03 16:13:36.590160 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:36.590135 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:13:36.596754 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:36.596716 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 16:08:36 +0000 UTC" deadline="2028-04-12 17:28:05.636299245 +0000 UTC" Nov 03 16:13:36.596754 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:36.596754 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12625h14m29.039549284s" Nov 03 16:13:36.612389 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:36.612359 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:13:36.699810 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:36.699754 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:13:36.725596 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:36.725565 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:13:36.728156 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:36.728136 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal" Nov 03 16:13:36.751092 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:36.751057 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:13:38.101319 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.101287 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:13:38.455990 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.455964 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:13:38.475945 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.475917 2577 apiserver.go:52] "Watching apiserver" Nov 03 16:13:38.499783 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.499756 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 16:13:38.512585 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.512520 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 16:13:38.512946 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.512921 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-bflh2","openshift-multus/multus-additional-cni-plugins-r6mtg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal","openshift-multus/multus-kmfdd","openshift-multus/network-metrics-daemon-n7hw7","openshift-dns/node-resolver-zfshj","openshift-network-diagnostics/network-check-target-5fq8q","openshift-network-operator/iptables-alerter-g5w4j","kube-system/global-pull-secret-syncer-p7vvq","kube-system/konnectivity-agent-zmzv6","kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal","openshift-cluster-node-tuning-operator/tuned-2kl7n","openshift-ovn-kubernetes/ovnkube-node-gvrm4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g"] Nov 03 16:13:38.515624 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.515592 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zmzv6" Nov 03 16:13:38.518174 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.517932 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bflh2" Nov 03 16:13:38.520298 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.520275 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.522908 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.522538 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.525184 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.525163 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:38.525298 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:38.525243 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7hw7" podUID="55767581-2059-49bf-8dd0-69e31d5adb83" Nov 03 16:13:38.527951 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.527928 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zfshj" Nov 03 16:13:38.530282 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.530256 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g5w4j" Nov 03 16:13:38.532634 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.532612 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.537678 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.535560 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.537678 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.536371 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2k4df\"" Nov 03 16:13:38.537678 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.536379 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q9tss\"" Nov 03 16:13:38.537678 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.536463 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 16:13:38.537678 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.537469 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hxhwt\"" Nov 03 16:13:38.538019 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.537895 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 16:13:38.538074 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.538020 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 16:13:38.538119 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.538086 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 16:13:38.538167 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.538158 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 16:13:38.538360 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.538343 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.538636 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.538615 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 16:13:38.541058 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.541033 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:38.541143 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:38.541105 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fq8q" podUID="110bff76-1581-4379-8468-441a3a81dea2" Nov 03 16:13:38.543722 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.543685 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:38.543814 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:38.543780 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7vvq" podUID="8c96f345-2cda-4554-bef3-be416dc50225" Nov 03 16:13:38.548935 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.548906 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 16:13:38.549055 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.548996 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 16:13:38.549533 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.549503 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n246q\"" Nov 03 16:13:38.549681 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.549656 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 16:13:38.549831 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.549815 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 16:13:38.549925 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.549886 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 16:13:38.550748 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.550326 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 16:13:38.556380 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.556223 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 16:13:38.556380 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.556305 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b74pv" Nov 03 16:13:38.574802 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.574759 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b74pv" Nov 03 16:13:38.584977 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.584910 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal" event={"ID":"9d137c53e9ec4edb6f2bb3a754f3bb53","Type":"ContainerStarted","Data":"cd11594e79cefb0bb847b42f8639cfaf6757665400c707c490511b666e78a220"} Nov 03 16:13:38.586196 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.586166 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" event={"ID":"10d52772a6531c8bd8223292030e42c2","Type":"ContainerStarted","Data":"2c6d7e35341f617447534ea331414166e75c5c6220a636690c5e2db938cc4ad7"} Nov 03 16:13:38.601568 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.601536 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 16:13:38.613035 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.612943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-device-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.613035 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-multus-cni-dir\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.613303 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613074 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7731b228-5421-4121-9228-1f19a850ef9a-cni-binary-copy\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.613303 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613123 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-hostroot\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.613303 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613153 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-sysctl-conf\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.613303 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613196 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-systemd\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.613303 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613220 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7f5be2e-c57e-432f-a5bb-144709f3b492-tmp\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.613303 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613244 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/812067ca-eef5-4ab1-a24e-61796ac0721b-host\") pod \"node-ca-bflh2\" (UID: \"812067ca-eef5-4ab1-a24e-61796ac0721b\") " pod="openshift-image-registry/node-ca-bflh2" Nov 03 16:13:38.613303 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613278 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-multus-conf-dir\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.613648 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613305 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jjxv\" (UniqueName: \"kubernetes.io/projected/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd-kube-api-access-5jjxv\") pod \"iptables-alerter-g5w4j\" (UID: \"ec0f2c68-a3bf-4475-ad0d-43346dbd32fd\") " pod="openshift-network-operator/iptables-alerter-g5w4j" Nov 03 16:13:38.613648 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613334 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cbd4c91-69f0-47be-94be-e451d44b4992-hosts-file\") pod \"node-resolver-zfshj\" (UID: \"3cbd4c91-69f0-47be-94be-e451d44b4992\") " pod="openshift-dns/node-resolver-zfshj" Nov 03 16:13:38.613648 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfv4w\" (UniqueName: \"kubernetes.io/projected/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-kube-api-access-cfv4w\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.613648 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613386 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-var-lib-cni-bin\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.613648 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613409 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-var-lib-kubelet\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.613648 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613492 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1566c753-13b1-4495-9268-05e3676d82bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.613648 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613521 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9slx\" (UniqueName: \"kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx\") pod \"network-check-target-5fq8q\" (UID: \"110bff76-1581-4379-8468-441a3a81dea2\") " pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:38.613648 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613546 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.613648 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613569 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-var-lib-cni-multus\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.613648 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613594 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-socket-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.613648 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613619 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-run-k8s-cni-cncf-io\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.613648 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613644 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-run\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.614223 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613666 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-tuned\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.614223 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613691 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:38.614223 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613743 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.614223 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613776 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-ovnkube-config\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.614223 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613856 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-sys\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.614223 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-lib-modules\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.614223 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613909 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9hhm\" (UniqueName: \"kubernetes.io/projected/a7f5be2e-c57e-432f-a5bb-144709f3b492-kube-api-access-j9hhm\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.614223 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613938 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-systemd-units\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.614223 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:38.614223 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.613989 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn2hk\" (UniqueName: \"kubernetes.io/projected/812067ca-eef5-4ab1-a24e-61796ac0721b-kube-api-access-hn2hk\") pod \"node-ca-bflh2\" (UID: \"812067ca-eef5-4ab1-a24e-61796ac0721b\") " pod="openshift-image-registry/node-ca-bflh2" Nov 03 16:13:38.614223 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614014 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-etc-selinux\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.614223 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614041 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.614223 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614070 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-registration-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.614223 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614095 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-cnibin\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.614223 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614120 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7731b228-5421-4121-9228-1f19a850ef9a-multus-daemon-config\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.614223 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614144 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp88d\" (UniqueName: \"kubernetes.io/projected/7731b228-5421-4121-9228-1f19a850ef9a-kube-api-access-vp88d\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.615012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614170 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd-host-slash\") pod \"iptables-alerter-g5w4j\" (UID: \"ec0f2c68-a3bf-4475-ad0d-43346dbd32fd\") " pod="openshift-network-operator/iptables-alerter-g5w4j" Nov 03 16:13:38.615012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614203 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-run-ovn\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.615012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614228 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-run-systemd\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.615012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614252 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-log-socket\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.615012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614284 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c96f345-2cda-4554-bef3-be416dc50225-kubelet-config\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:38.615012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614309 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-sys-fs\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.615012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614344 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-run-multus-certs\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.615012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614371 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd-iptables-alerter-script\") pod \"iptables-alerter-g5w4j\" (UID: \"ec0f2c68-a3bf-4475-ad0d-43346dbd32fd\") " pod="openshift-network-operator/iptables-alerter-g5w4j" Nov 03 16:13:38.615012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614397 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-modprobe-d\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.615012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614532 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1566c753-13b1-4495-9268-05e3676d82bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.615012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614563 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/be60fb17-308e-420d-b9e0-d9a05c0c752f-konnectivity-ca\") pod \"konnectivity-agent-zmzv6\" (UID: \"be60fb17-308e-420d-b9e0-d9a05c0c752f\") " pod="kube-system/konnectivity-agent-zmzv6" Nov 03 16:13:38.615012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614588 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-ovnkube-script-lib\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.615012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614615 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjwvk\" (UniqueName: \"kubernetes.io/projected/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-kube-api-access-gjwvk\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.615012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614642 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1566c753-13b1-4495-9268-05e3676d82bd-system-cni-dir\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.615012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614668 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/be60fb17-308e-420d-b9e0-d9a05c0c752f-agent-certs\") pod \"konnectivity-agent-zmzv6\" (UID: \"be60fb17-308e-420d-b9e0-d9a05c0c752f\") " pod="kube-system/konnectivity-agent-zmzv6" Nov 03 16:13:38.615012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614710 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z647\" (UniqueName: \"kubernetes.io/projected/55767581-2059-49bf-8dd0-69e31d5adb83-kube-api-access-2z647\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:38.615761 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-etc-openvswitch\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.615761 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614773 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-cni-netd\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.615761 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614798 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/812067ca-eef5-4ab1-a24e-61796ac0721b-serviceca\") pod \"node-ca-bflh2\" (UID: \"812067ca-eef5-4ab1-a24e-61796ac0721b\") " pod="openshift-image-registry/node-ca-bflh2" Nov 03 16:13:38.615761 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-kubernetes\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.615761 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614849 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwcf4\" (UniqueName: \"kubernetes.io/projected/1566c753-13b1-4495-9268-05e3676d82bd-kube-api-access-vwcf4\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.615761 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614876 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-kubelet\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.615761 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614901 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-cni-bin\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.615761 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614926 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-sysconfig\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.615761 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c96f345-2cda-4554-bef3-be416dc50225-dbus\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:38.615761 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.614975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-system-cni-dir\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.615761 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.615001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-multus-socket-dir-parent\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.615761 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.615027 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-run-netns\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.615761 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.615061 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-sysctl-d\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.615761 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.615086 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-var-lib-kubelet\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.615761 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.615112 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1566c753-13b1-4495-9268-05e3676d82bd-cnibin\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.615761 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.615138 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-run-netns\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.616513 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.615164 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-var-lib-openvswitch\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.616513 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.615193 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-etc-kubernetes\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.616513 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.615224 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1566c753-13b1-4495-9268-05e3676d82bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.616513 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.615251 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4h7b\" (UniqueName: \"kubernetes.io/projected/3cbd4c91-69f0-47be-94be-e451d44b4992-kube-api-access-l4h7b\") pod \"node-resolver-zfshj\" (UID: \"3cbd4c91-69f0-47be-94be-e451d44b4992\") " pod="openshift-dns/node-resolver-zfshj" Nov 03 16:13:38.616513 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.615277 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-run-openvswitch\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.616513 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.615304 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-env-overrides\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.616513 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.615329 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-host\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.616513 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.615354 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1566c753-13b1-4495-9268-05e3676d82bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.616513 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.615380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cbd4c91-69f0-47be-94be-e451d44b4992-tmp-dir\") pod \"node-resolver-zfshj\" (UID: \"3cbd4c91-69f0-47be-94be-e451d44b4992\") " pod="openshift-dns/node-resolver-zfshj" Nov 03 16:13:38.616513 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.615405 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-slash\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.616513 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.615439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-node-log\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.616513 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.615467 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-ovn-node-metrics-cert\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.636275 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.636241 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 16:13:38.636461 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.636349 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:13:38.636538 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.636500 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 16:13:38.636598 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.636553 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 16:13:38.636598 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.636561 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6q76v\"" Nov 03 16:13:38.636598 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.636570 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 16:13:38.636777 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.636595 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 16:13:38.636777 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.636608 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 16:13:38.636777 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.636612 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 16:13:38.636777 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.636642 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 16:13:38.636777 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.636582 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 16:13:38.637012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.636807 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8mlmt\"" Nov 03 16:13:38.637012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.636817 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 16:13:38.637012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.636847 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:13:38.637158 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.637075 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 16:13:38.637158 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.637141 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5xwqx\"" Nov 03 16:13:38.637257 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.637233 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tn5d9\"" Nov 03 16:13:38.637355 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.637339 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 16:13:38.637422 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.637394 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tf624\"" Nov 03 16:13:38.637523 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.637509 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 16:13:38.716646 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.716553 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-ovnkube-script-lib\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.716646 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.716596 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjwvk\" (UniqueName: \"kubernetes.io/projected/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-kube-api-access-gjwvk\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.716646 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.716615 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1566c753-13b1-4495-9268-05e3676d82bd-system-cni-dir\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.716646 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.716639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/be60fb17-308e-420d-b9e0-d9a05c0c752f-agent-certs\") pod \"konnectivity-agent-zmzv6\" (UID: \"be60fb17-308e-420d-b9e0-d9a05c0c752f\") " pod="kube-system/konnectivity-agent-zmzv6" Nov 03 16:13:38.716980 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.716667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z647\" (UniqueName: \"kubernetes.io/projected/55767581-2059-49bf-8dd0-69e31d5adb83-kube-api-access-2z647\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:38.716980 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.716690 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-etc-openvswitch\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.716980 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.716728 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-cni-netd\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.716980 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.716753 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/812067ca-eef5-4ab1-a24e-61796ac0721b-serviceca\") pod \"node-ca-bflh2\" (UID: \"812067ca-eef5-4ab1-a24e-61796ac0721b\") " pod="openshift-image-registry/node-ca-bflh2" Nov 03 16:13:38.716980 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.716776 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-kubernetes\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.716980 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.716796 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwcf4\" (UniqueName: \"kubernetes.io/projected/1566c753-13b1-4495-9268-05e3676d82bd-kube-api-access-vwcf4\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.716980 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.716816 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-kubelet\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.716980 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.716831 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-cni-bin\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.716980 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.716853 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-sysconfig\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.716980 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.716875 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c96f345-2cda-4554-bef3-be416dc50225-dbus\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:38.716980 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.716897 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-system-cni-dir\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.716980 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.716921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-multus-socket-dir-parent\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.716980 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.716947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-run-netns\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.716980 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.716982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-sysctl-d\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.717583 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-var-lib-kubelet\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.717583 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717029 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1566c753-13b1-4495-9268-05e3676d82bd-cnibin\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.717583 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717054 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-run-netns\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.717583 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717078 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-var-lib-openvswitch\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.717583 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717102 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-etc-kubernetes\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.717583 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1566c753-13b1-4495-9268-05e3676d82bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.717583 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4h7b\" (UniqueName: \"kubernetes.io/projected/3cbd4c91-69f0-47be-94be-e451d44b4992-kube-api-access-l4h7b\") pod \"node-resolver-zfshj\" (UID: \"3cbd4c91-69f0-47be-94be-e451d44b4992\") " pod="openshift-dns/node-resolver-zfshj" Nov 03 16:13:38.717583 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-run-openvswitch\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.717583 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717211 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-env-overrides\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.717583 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717233 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-host\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.717583 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717262 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1566c753-13b1-4495-9268-05e3676d82bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.717583 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717286 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cbd4c91-69f0-47be-94be-e451d44b4992-tmp-dir\") pod \"node-resolver-zfshj\" (UID: \"3cbd4c91-69f0-47be-94be-e451d44b4992\") " pod="openshift-dns/node-resolver-zfshj" Nov 03 16:13:38.717583 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717311 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-slash\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.717583 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717336 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-node-log\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.717583 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717359 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-ovn-node-metrics-cert\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.717583 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717375 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-device-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.717583 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717412 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-multus-cni-dir\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.719691 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717446 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7731b228-5421-4121-9228-1f19a850ef9a-cni-binary-copy\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.719691 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717494 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-hostroot\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.719691 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717517 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-sysctl-conf\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.719691 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717539 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-systemd\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.719691 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717562 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7f5be2e-c57e-432f-a5bb-144709f3b492-tmp\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.719691 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/812067ca-eef5-4ab1-a24e-61796ac0721b-host\") pod \"node-ca-bflh2\" (UID: \"812067ca-eef5-4ab1-a24e-61796ac0721b\") " pod="openshift-image-registry/node-ca-bflh2" Nov 03 16:13:38.719691 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717618 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-multus-conf-dir\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.719691 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jjxv\" (UniqueName: \"kubernetes.io/projected/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd-kube-api-access-5jjxv\") pod \"iptables-alerter-g5w4j\" (UID: \"ec0f2c68-a3bf-4475-ad0d-43346dbd32fd\") " pod="openshift-network-operator/iptables-alerter-g5w4j" Nov 03 16:13:38.719691 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717681 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cbd4c91-69f0-47be-94be-e451d44b4992-hosts-file\") pod \"node-resolver-zfshj\" (UID: \"3cbd4c91-69f0-47be-94be-e451d44b4992\") " pod="openshift-dns/node-resolver-zfshj" Nov 03 16:13:38.719691 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfv4w\" (UniqueName: \"kubernetes.io/projected/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-kube-api-access-cfv4w\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.719691 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717767 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-var-lib-cni-bin\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.719691 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-var-lib-kubelet\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.719691 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717821 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1566c753-13b1-4495-9268-05e3676d82bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.719691 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717828 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-var-lib-openvswitch\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.719691 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717845 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9slx\" (UniqueName: \"kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx\") pod \"network-check-target-5fq8q\" (UID: \"110bff76-1581-4379-8468-441a3a81dea2\") " pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:38.719691 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717873 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.719691 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717907 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-var-lib-cni-multus\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.720528 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.717948 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-socket-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.720528 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718030 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-run-k8s-cni-cncf-io\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.720528 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718074 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-run\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.720528 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718107 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-tuned\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.720528 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:38.720528 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718179 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.720528 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-ovnkube-config\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.720528 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718224 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-sys\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.720528 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-lib-modules\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.720528 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718276 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9hhm\" (UniqueName: \"kubernetes.io/projected/a7f5be2e-c57e-432f-a5bb-144709f3b492-kube-api-access-j9hhm\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.720528 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718314 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-systemd-units\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.720528 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718342 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:38.720528 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718366 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn2hk\" (UniqueName: \"kubernetes.io/projected/812067ca-eef5-4ab1-a24e-61796ac0721b-kube-api-access-hn2hk\") pod \"node-ca-bflh2\" (UID: \"812067ca-eef5-4ab1-a24e-61796ac0721b\") " pod="openshift-image-registry/node-ca-bflh2" Nov 03 16:13:38.720528 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718388 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-etc-selinux\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.720528 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718404 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.720528 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718438 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-registration-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.720528 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718461 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-cnibin\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.721354 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718552 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7731b228-5421-4121-9228-1f19a850ef9a-multus-daemon-config\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.721354 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp88d\" (UniqueName: \"kubernetes.io/projected/7731b228-5421-4121-9228-1f19a850ef9a-kube-api-access-vp88d\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.721354 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718609 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd-host-slash\") pod \"iptables-alerter-g5w4j\" (UID: \"ec0f2c68-a3bf-4475-ad0d-43346dbd32fd\") " pod="openshift-network-operator/iptables-alerter-g5w4j" Nov 03 16:13:38.721354 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-run-ovn\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.721354 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718667 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-var-lib-cni-bin\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.721354 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718688 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-run-systemd\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.721354 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718746 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-run-systemd\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.721354 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718746 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-log-socket\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.721354 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718743 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-device-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.721354 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c96f345-2cda-4554-bef3-be416dc50225-kubelet-config\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:38.721354 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718801 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-sys\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.721354 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718830 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-sys-fs\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.721354 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718864 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-run-multus-certs\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.721354 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718882 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-etc-kubernetes\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.721354 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd-iptables-alerter-script\") pod \"iptables-alerter-g5w4j\" (UID: \"ec0f2c68-a3bf-4475-ad0d-43346dbd32fd\") " pod="openshift-network-operator/iptables-alerter-g5w4j" Nov 03 16:13:38.721354 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718951 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-kubelet\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.721354 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718957 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-modprobe-d\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.721354 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.718985 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-lib-modules\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.722271 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.719014 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1566c753-13b1-4495-9268-05e3676d82bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.722271 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.719047 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/be60fb17-308e-420d-b9e0-d9a05c0c752f-konnectivity-ca\") pod \"konnectivity-agent-zmzv6\" (UID: \"be60fb17-308e-420d-b9e0-d9a05c0c752f\") " pod="kube-system/konnectivity-agent-zmzv6" Nov 03 16:13:38.722271 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.719062 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-systemd-units\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.722271 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.719105 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-modprobe-d\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.722271 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.719142 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-etc-openvswitch\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.722271 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.719217 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-var-lib-cni-multus\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.722271 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.719259 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-cni-netd\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.722271 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:38.719433 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:38.722271 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.719789 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-var-lib-kubelet\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.722271 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.719829 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-run-k8s-cni-cncf-io\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.722271 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.719848 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-run-openvswitch\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.722271 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.719860 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-sysconfig\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.722271 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.719901 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-run-multus-certs\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.722271 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.719955 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-system-cni-dir\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.722271 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.719974 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-var-lib-kubelet\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.722271 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.719985 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-systemd\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.722271 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.720075 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c96f345-2cda-4554-bef3-be416dc50225-dbus\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:38.722271 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.720104 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-sysctl-conf\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.724512 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.720094 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-multus-cni-dir\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.724512 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.720133 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-host\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.724512 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.720158 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-run-netns\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.724512 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:38.720203 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:38.724512 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.720268 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-sysctl-d\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.724512 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.720330 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-registration-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.724512 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:38.720391 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs podName:55767581-2059-49bf-8dd0-69e31d5adb83 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:39.220346359 +0000 UTC m=+3.146251228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs") pod "network-metrics-daemon-n7hw7" (UID: "55767581-2059-49bf-8dd0-69e31d5adb83") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:38.724512 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.720456 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.724512 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.720531 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1566c753-13b1-4495-9268-05e3676d82bd-cnibin\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.724512 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.720675 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-kubernetes\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.724512 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.720746 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1566c753-13b1-4495-9268-05e3676d82bd-system-cni-dir\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.724512 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.720789 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/812067ca-eef5-4ab1-a24e-61796ac0721b-host\") pod \"node-ca-bflh2\" (UID: \"812067ca-eef5-4ab1-a24e-61796ac0721b\") " pod="openshift-image-registry/node-ca-bflh2" Nov 03 16:13:38.724512 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.720832 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-cnibin\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.724512 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.721254 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.724512 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:38.721414 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret podName:8c96f345-2cda-4554-bef3-be416dc50225 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:39.221390942 +0000 UTC m=+3.147295804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret") pod "global-pull-secret-syncer-p7vvq" (UID: "8c96f345-2cda-4554-bef3-be416dc50225") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:38.724512 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.721472 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-slash\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.724512 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.721639 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-cni-bin\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.725348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.721745 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-node-log\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.725348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.721812 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-log-socket\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.725348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.721874 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-multus-conf-dir\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.725348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.721885 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cbd4c91-69f0-47be-94be-e451d44b4992-hosts-file\") pod \"node-resolver-zfshj\" (UID: \"3cbd4c91-69f0-47be-94be-e451d44b4992\") " pod="openshift-dns/node-resolver-zfshj" Nov 03 16:13:38.725348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.721934 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-run\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.725348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.721958 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c96f345-2cda-4554-bef3-be416dc50225-kubelet-config\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:38.725348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.721990 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-etc-selinux\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.725348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.722096 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.725348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.722118 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-sys-fs\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.725348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.722157 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-hostroot\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.725348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.722192 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd-host-slash\") pod \"iptables-alerter-g5w4j\" (UID: \"ec0f2c68-a3bf-4475-ad0d-43346dbd32fd\") " pod="openshift-network-operator/iptables-alerter-g5w4j" Nov 03 16:13:38.725348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.722583 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-run-ovn\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.725348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.722660 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-multus-socket-dir-parent\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.725348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.722675 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-run-netns\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.725348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.722877 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-ovnkube-script-lib\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.725348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.723435 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-ovnkube-config\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.725348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.723480 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-env-overrides\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.725348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.723675 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1566c753-13b1-4495-9268-05e3676d82bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.726115 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.723843 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cbd4c91-69f0-47be-94be-e451d44b4992-tmp-dir\") pod \"node-resolver-zfshj\" (UID: \"3cbd4c91-69f0-47be-94be-e451d44b4992\") " pod="openshift-dns/node-resolver-zfshj" Nov 03 16:13:38.726115 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.723999 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/812067ca-eef5-4ab1-a24e-61796ac0721b-serviceca\") pod \"node-ca-bflh2\" (UID: \"812067ca-eef5-4ab1-a24e-61796ac0721b\") " pod="openshift-image-registry/node-ca-bflh2" Nov 03 16:13:38.726115 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.725065 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7731b228-5421-4121-9228-1f19a850ef9a-multus-daemon-config\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.726115 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.725097 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7731b228-5421-4121-9228-1f19a850ef9a-cni-binary-copy\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.726115 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.725235 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-socket-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.726115 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.725242 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 16:13:38.726115 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.725243 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd-iptables-alerter-script\") pod \"iptables-alerter-g5w4j\" (UID: \"ec0f2c68-a3bf-4475-ad0d-43346dbd32fd\") " pod="openshift-network-operator/iptables-alerter-g5w4j" Nov 03 16:13:38.726115 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.725610 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1566c753-13b1-4495-9268-05e3676d82bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.726115 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.725652 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1566c753-13b1-4495-9268-05e3676d82bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.726115 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.725710 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/be60fb17-308e-420d-b9e0-d9a05c0c752f-konnectivity-ca\") pod \"konnectivity-agent-zmzv6\" (UID: \"be60fb17-308e-420d-b9e0-d9a05c0c752f\") " pod="kube-system/konnectivity-agent-zmzv6" Nov 03 16:13:38.726576 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.726337 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1566c753-13b1-4495-9268-05e3676d82bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.730338 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.730308 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7f5be2e-c57e-432f-a5bb-144709f3b492-tmp\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.730466 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.730415 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/be60fb17-308e-420d-b9e0-d9a05c0c752f-agent-certs\") pod \"konnectivity-agent-zmzv6\" (UID: \"be60fb17-308e-420d-b9e0-d9a05c0c752f\") " pod="kube-system/konnectivity-agent-zmzv6" Nov 03 16:13:38.731783 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.731760 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-tuned\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.733882 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.733860 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-ovn-node-metrics-cert\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.747853 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:38.747792 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:13:38.747853 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:38.747827 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:13:38.747853 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:38.747841 2577 projected.go:196] Error preparing data for projected volume kube-api-access-l9slx for pod openshift-network-diagnostics/network-check-target-5fq8q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:38.748140 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:38.747903 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx podName:110bff76-1581-4379-8468-441a3a81dea2 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:39.247885337 +0000 UTC m=+3.173790207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l9slx" (UniqueName: "kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx") pod "network-check-target-5fq8q" (UID: "110bff76-1581-4379-8468-441a3a81dea2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:38.748208 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.748157 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjwvk\" (UniqueName: \"kubernetes.io/projected/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-kube-api-access-gjwvk\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.748643 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.748618 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9hhm\" (UniqueName: \"kubernetes.io/projected/a7f5be2e-c57e-432f-a5bb-144709f3b492-kube-api-access-j9hhm\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.750103 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.750044 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn2hk\" (UniqueName: \"kubernetes.io/projected/812067ca-eef5-4ab1-a24e-61796ac0721b-kube-api-access-hn2hk\") pod \"node-ca-bflh2\" (UID: \"812067ca-eef5-4ab1-a24e-61796ac0721b\") " pod="openshift-image-registry/node-ca-bflh2" Nov 03 16:13:38.752108 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.752069 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfv4w\" (UniqueName: \"kubernetes.io/projected/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-kube-api-access-cfv4w\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.752656 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.752633 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp88d\" (UniqueName: \"kubernetes.io/projected/7731b228-5421-4121-9228-1f19a850ef9a-kube-api-access-vp88d\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.752773 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.752659 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwcf4\" (UniqueName: \"kubernetes.io/projected/1566c753-13b1-4495-9268-05e3676d82bd-kube-api-access-vwcf4\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.753870 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.753843 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jjxv\" (UniqueName: \"kubernetes.io/projected/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd-kube-api-access-5jjxv\") pod \"iptables-alerter-g5w4j\" (UID: \"ec0f2c68-a3bf-4475-ad0d-43346dbd32fd\") " pod="openshift-network-operator/iptables-alerter-g5w4j" Nov 03 16:13:38.755573 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.755544 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4h7b\" (UniqueName: \"kubernetes.io/projected/3cbd4c91-69f0-47be-94be-e451d44b4992-kube-api-access-l4h7b\") pod \"node-resolver-zfshj\" (UID: \"3cbd4c91-69f0-47be-94be-e451d44b4992\") " pod="openshift-dns/node-resolver-zfshj" Nov 03 16:13:38.755687 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.755593 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z647\" (UniqueName: \"kubernetes.io/projected/55767581-2059-49bf-8dd0-69e31d5adb83-kube-api-access-2z647\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:38.831681 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.831640 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zmzv6" Nov 03 16:13:38.840782 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.840745 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bflh2" Nov 03 16:13:38.844742 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:38.844687 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe60fb17_308e_420d_b9e0_d9a05c0c752f.slice/crio-2ba6d971399525cdd8e1815130a771c92ed0d3452825a191e0dadf1428bd25ff WatchSource:0}: Error finding container 2ba6d971399525cdd8e1815130a771c92ed0d3452825a191e0dadf1428bd25ff: Status 404 returned error can't find the container with id 2ba6d971399525cdd8e1815130a771c92ed0d3452825a191e0dadf1428bd25ff Nov 03 16:13:38.851789 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:38.851757 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812067ca_eef5_4ab1_a24e_61796ac0721b.slice/crio-60a1c67ee846d8eb71f014ba0b3afff8b717a1c232f4e2abacc8bd352df6038d WatchSource:0}: Error finding container 60a1c67ee846d8eb71f014ba0b3afff8b717a1c232f4e2abacc8bd352df6038d: Status 404 returned error can't find the container with id 60a1c67ee846d8eb71f014ba0b3afff8b717a1c232f4e2abacc8bd352df6038d Nov 03 16:13:38.858443 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.858415 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kmfdd" Nov 03 16:13:38.858605 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.858584 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:13:38.867504 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.867477 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zfshj" Nov 03 16:13:38.874024 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.873999 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g5w4j" Nov 03 16:13:38.878551 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:38.877970 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ce380e_3ddd_42bd_867c_9bba84f2bab4.slice/crio-60a6fb6a35818963c1f0b23463654b6788fe8550e94e88e2690ab09d530769ac WatchSource:0}: Error finding container 60a6fb6a35818963c1f0b23463654b6788fe8550e94e88e2690ab09d530769ac: Status 404 returned error can't find the container with id 60a6fb6a35818963c1f0b23463654b6788fe8550e94e88e2690ab09d530769ac Nov 03 16:13:38.878551 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:38.878432 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7731b228_5421_4121_9228_1f19a850ef9a.slice/crio-33f6cb62246e179f45746f693ebceda4e77fdf4ccc88bf16da7c137ce2d02e70 WatchSource:0}: Error finding container 33f6cb62246e179f45746f693ebceda4e77fdf4ccc88bf16da7c137ce2d02e70: Status 404 returned error can't find the container with id 33f6cb62246e179f45746f693ebceda4e77fdf4ccc88bf16da7c137ce2d02e70 Nov 03 16:13:38.881787 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.881762 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:13:38.882311 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:38.882279 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cbd4c91_69f0_47be_94be_e451d44b4992.slice/crio-8d99312e03eaf44c940fea1c6bd9e48ab0f5fdf296cf48a30b96f996e257b1c9 WatchSource:0}: Error finding container 8d99312e03eaf44c940fea1c6bd9e48ab0f5fdf296cf48a30b96f996e257b1c9: Status 404 returned error can't find the container with id 8d99312e03eaf44c940fea1c6bd9e48ab0f5fdf296cf48a30b96f996e257b1c9 Nov 03 16:13:38.885291 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:38.885271 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec0f2c68_a3bf_4475_ad0d_43346dbd32fd.slice/crio-9f8c99e401fa37c70cbaa90bad16117ee66798b83394f66d5c63cd769043fb74 WatchSource:0}: Error finding container 9f8c99e401fa37c70cbaa90bad16117ee66798b83394f66d5c63cd769043fb74: Status 404 returned error can't find the container with id 9f8c99e401fa37c70cbaa90bad16117ee66798b83394f66d5c63cd769043fb74 Nov 03 16:13:38.893749 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:38.893722 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f5be2e_c57e_432f_a5bb_144709f3b492.slice/crio-0d94d4680f57ca39ae6cde2be2c7e1fef83d64026b2196f253c35331d8ddb538 WatchSource:0}: Error finding container 0d94d4680f57ca39ae6cde2be2c7e1fef83d64026b2196f253c35331d8ddb538: Status 404 returned error can't find the container with id 0d94d4680f57ca39ae6cde2be2c7e1fef83d64026b2196f253c35331d8ddb538 Nov 03 16:13:38.906983 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.906952 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:13:38.919457 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:38.919414 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1566c753_13b1_4495_9268_05e3676d82bd.slice/crio-da3d72656b51ca863a1d7ed72013624774cd07b382ce9eaf834827431b293899 WatchSource:0}: Error finding container da3d72656b51ca863a1d7ed72013624774cd07b382ce9eaf834827431b293899: Status 404 returned error can't find the container with id da3d72656b51ca863a1d7ed72013624774cd07b382ce9eaf834827431b293899 Nov 03 16:13:38.920516 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:38.920463 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:38.932239 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:13:38.932208 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec02fd2_78a5_4813_b6e4_4edefa217fe1.slice/crio-c0d5f41f36ea2f8106c2bd9f0ce12c720e987eddbe6cde2bbc673bcd2a235f67 WatchSource:0}: Error finding container c0d5f41f36ea2f8106c2bd9f0ce12c720e987eddbe6cde2bbc673bcd2a235f67: Status 404 returned error can't find the container with id c0d5f41f36ea2f8106c2bd9f0ce12c720e987eddbe6cde2bbc673bcd2a235f67 Nov 03 16:13:39.225753 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.223018 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:39.225753 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.223309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:39.225753 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:39.223454 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:39.225753 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:39.223517 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret podName:8c96f345-2cda-4554-bef3-be416dc50225 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:40.223500563 +0000 UTC m=+4.149405417 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret") pod "global-pull-secret-syncer-p7vvq" (UID: "8c96f345-2cda-4554-bef3-be416dc50225") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:39.225753 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:39.223904 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:39.225753 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:39.223960 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs podName:55767581-2059-49bf-8dd0-69e31d5adb83 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:40.223945534 +0000 UTC m=+4.149850399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs") pod "network-metrics-daemon-n7hw7" (UID: "55767581-2059-49bf-8dd0-69e31d5adb83") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:39.323903 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.323864 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9slx\" (UniqueName: \"kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx\") pod \"network-check-target-5fq8q\" (UID: \"110bff76-1581-4379-8468-441a3a81dea2\") " pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:39.324084 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:39.324069 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:13:39.324147 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:39.324092 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:13:39.324147 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:39.324104 2577 projected.go:196] Error preparing data for projected volume kube-api-access-l9slx for pod openshift-network-diagnostics/network-check-target-5fq8q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:39.324244 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:39.324164 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx podName:110bff76-1581-4379-8468-441a3a81dea2 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:40.324145922 +0000 UTC m=+4.250050790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9slx" (UniqueName: "kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx") pod "network-check-target-5fq8q" (UID: "110bff76-1581-4379-8468-441a3a81dea2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:39.580065 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.579863 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 16:08:38 +0000 UTC" deadline="2028-04-01 20:49:35.198182054 +0000 UTC" Nov 03 16:13:39.580065 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.579934 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12364h35m55.618252606s" Nov 03 16:13:39.597891 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.597846 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal" event={"ID":"9d137c53e9ec4edb6f2bb3a754f3bb53","Type":"ContainerStarted","Data":"fbf3dfb75efd2778a51fb0f04629ba8eec1f077924de089eec69bed67d1c2cd6"} Nov 03 16:13:39.618591 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.617837 2577 generic.go:356] "Generic (PLEG): container finished" podID="10d52772a6531c8bd8223292030e42c2" containerID="e2963ede6517632a7b5156383b0cd5e7accc98704b47c2ce6afbd4da72c153d0" exitCode=0 Nov 03 16:13:39.618591 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.617922 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" event={"ID":"10d52772a6531c8bd8223292030e42c2","Type":"ContainerDied","Data":"e2963ede6517632a7b5156383b0cd5e7accc98704b47c2ce6afbd4da72c153d0"} Nov 03 16:13:39.621243 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.621179 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:13:39.624022 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.623932 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r6mtg" event={"ID":"1566c753-13b1-4495-9268-05e3676d82bd","Type":"ContainerStarted","Data":"da3d72656b51ca863a1d7ed72013624774cd07b382ce9eaf834827431b293899"} Nov 03 16:13:39.629261 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.629161 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g5w4j" event={"ID":"ec0f2c68-a3bf-4475-ad0d-43346dbd32fd","Type":"ContainerStarted","Data":"9f8c99e401fa37c70cbaa90bad16117ee66798b83394f66d5c63cd769043fb74"} Nov 03 16:13:39.638070 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.637982 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zfshj" event={"ID":"3cbd4c91-69f0-47be-94be-e451d44b4992","Type":"ContainerStarted","Data":"8d99312e03eaf44c940fea1c6bd9e48ab0f5fdf296cf48a30b96f996e257b1c9"} Nov 03 16:13:39.649644 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.649541 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" event={"ID":"a3ce380e-3ddd-42bd-867c-9bba84f2bab4","Type":"ContainerStarted","Data":"60a6fb6a35818963c1f0b23463654b6788fe8550e94e88e2690ab09d530769ac"} Nov 03 16:13:39.651913 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.651798 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zmzv6" event={"ID":"be60fb17-308e-420d-b9e0-d9a05c0c752f","Type":"ContainerStarted","Data":"2ba6d971399525cdd8e1815130a771c92ed0d3452825a191e0dadf1428bd25ff"} Nov 03 16:13:39.656368 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.656248 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" event={"ID":"7ec02fd2-78a5-4813-b6e4-4edefa217fe1","Type":"ContainerStarted","Data":"c0d5f41f36ea2f8106c2bd9f0ce12c720e987eddbe6cde2bbc673bcd2a235f67"} Nov 03 16:13:39.663988 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.663952 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" event={"ID":"a7f5be2e-c57e-432f-a5bb-144709f3b492","Type":"ContainerStarted","Data":"0d94d4680f57ca39ae6cde2be2c7e1fef83d64026b2196f253c35331d8ddb538"} Nov 03 16:13:39.684192 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.684152 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kmfdd" event={"ID":"7731b228-5421-4121-9228-1f19a850ef9a","Type":"ContainerStarted","Data":"33f6cb62246e179f45746f693ebceda4e77fdf4ccc88bf16da7c137ce2d02e70"} Nov 03 16:13:39.686882 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.686084 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal" podStartSLOduration=2.686068814 podStartE2EDuration="2.686068814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:39.642410103 +0000 UTC m=+3.568314977" watchObservedRunningTime="2026-11-03 16:13:39.686068814 +0000 UTC m=+3.611973688" Nov 03 16:13:39.692857 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:39.692589 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bflh2" event={"ID":"812067ca-eef5-4ab1-a24e-61796ac0721b","Type":"ContainerStarted","Data":"60a1c67ee846d8eb71f014ba0b3afff8b717a1c232f4e2abacc8bd352df6038d"} Nov 03 16:13:40.229742 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:40.229634 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:40.229742 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:40.229714 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:40.229971 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:40.229885 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:40.229971 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:40.229948 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret podName:8c96f345-2cda-4554-bef3-be416dc50225 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:42.229930104 +0000 UTC m=+6.155834977 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret") pod "global-pull-secret-syncer-p7vvq" (UID: "8c96f345-2cda-4554-bef3-be416dc50225") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:40.230079 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:40.230021 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:40.230079 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:40.230055 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs podName:55767581-2059-49bf-8dd0-69e31d5adb83 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:42.23004353 +0000 UTC m=+6.155948395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs") pod "network-metrics-daemon-n7hw7" (UID: "55767581-2059-49bf-8dd0-69e31d5adb83") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:40.331410 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:40.330672 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9slx\" (UniqueName: \"kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx\") pod \"network-check-target-5fq8q\" (UID: \"110bff76-1581-4379-8468-441a3a81dea2\") " pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:40.331410 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:40.330906 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:13:40.331410 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:40.330925 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:13:40.331410 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:40.330939 2577 projected.go:196] Error preparing data for projected volume kube-api-access-l9slx for pod openshift-network-diagnostics/network-check-target-5fq8q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:40.331410 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:40.330997 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx podName:110bff76-1581-4379-8468-441a3a81dea2 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:42.330980842 +0000 UTC m=+6.256885705 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9slx" (UniqueName: "kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx") pod "network-check-target-5fq8q" (UID: "110bff76-1581-4379-8468-441a3a81dea2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:40.588667 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:40.587861 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:40.588667 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:40.588010 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7hw7" podUID="55767581-2059-49bf-8dd0-69e31d5adb83" Nov 03 16:13:40.588667 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:40.588429 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:40.588667 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:40.588526 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fq8q" podUID="110bff76-1581-4379-8468-441a3a81dea2" Nov 03 16:13:40.588667 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:40.588619 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 16:08:38 +0000 UTC" deadline="2028-07-17 06:28:12.03349958 +0000 UTC" Nov 03 16:13:40.588667 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:40.588641 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14918h14m31.444862459s" Nov 03 16:13:40.589523 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:40.589492 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:40.589641 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:40.589600 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7vvq" podUID="8c96f345-2cda-4554-bef3-be416dc50225" Nov 03 16:13:40.720290 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:40.720216 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" event={"ID":"10d52772a6531c8bd8223292030e42c2","Type":"ContainerStarted","Data":"2669f9756dffea804eb5f9a02d0538ed5c338c00266f588522d1668afca26f1e"} Nov 03 16:13:41.731601 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:41.731505 2577 generic.go:356] "Generic (PLEG): container finished" podID="1566c753-13b1-4495-9268-05e3676d82bd" containerID="9d6f3251807b378afdba565cb46127634bbd4ff94fbdae491344df215a477101" exitCode=0 Nov 03 16:13:41.732651 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:41.732618 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r6mtg" event={"ID":"1566c753-13b1-4495-9268-05e3676d82bd","Type":"ContainerDied","Data":"9d6f3251807b378afdba565cb46127634bbd4ff94fbdae491344df215a477101"} Nov 03 16:13:41.807956 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:41.806677 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" podStartSLOduration=4.80665542 podStartE2EDuration="4.80665542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:40.77149463 +0000 UTC m=+4.697399504" watchObservedRunningTime="2026-11-03 16:13:41.80665542 +0000 UTC m=+5.732560298" Nov 03 16:13:42.248155 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:42.248113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:42.248385 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:42.248172 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:42.248385 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:42.248343 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:42.248557 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:42.248408 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret podName:8c96f345-2cda-4554-bef3-be416dc50225 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:46.248390668 +0000 UTC m=+10.174295534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret") pod "global-pull-secret-syncer-p7vvq" (UID: "8c96f345-2cda-4554-bef3-be416dc50225") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:42.252063 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:42.251901 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:42.252063 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:42.252005 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs podName:55767581-2059-49bf-8dd0-69e31d5adb83 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:46.251983282 +0000 UTC m=+10.177888157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs") pod "network-metrics-daemon-n7hw7" (UID: "55767581-2059-49bf-8dd0-69e31d5adb83") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:42.348680 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:42.348638 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9slx\" (UniqueName: \"kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx\") pod \"network-check-target-5fq8q\" (UID: \"110bff76-1581-4379-8468-441a3a81dea2\") " pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:42.349372 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:42.348894 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:13:42.349372 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:42.348920 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:13:42.349372 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:42.348934 2577 projected.go:196] Error preparing data for projected volume kube-api-access-l9slx for pod openshift-network-diagnostics/network-check-target-5fq8q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:42.349372 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:42.348996 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx podName:110bff76-1581-4379-8468-441a3a81dea2 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:46.348978199 +0000 UTC m=+10.274883057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9slx" (UniqueName: "kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx") pod "network-check-target-5fq8q" (UID: "110bff76-1581-4379-8468-441a3a81dea2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:42.582930 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:42.582849 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:42.582930 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:42.582849 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:42.583165 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:42.583091 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7hw7" podUID="55767581-2059-49bf-8dd0-69e31d5adb83" Nov 03 16:13:42.583463 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:42.583444 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:42.583573 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:42.583544 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fq8q" podUID="110bff76-1581-4379-8468-441a3a81dea2" Nov 03 16:13:42.583660 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:42.583640 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7vvq" podUID="8c96f345-2cda-4554-bef3-be416dc50225" Nov 03 16:13:43.744798 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:43.743974 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zmzv6" event={"ID":"be60fb17-308e-420d-b9e0-d9a05c0c752f","Type":"ContainerStarted","Data":"fb22f8bc07c80171a0ca7aa8299be063f161f5b57b08b3cc5b9f942d1f156a56"} Nov 03 16:13:44.583093 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:44.583048 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:44.583295 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:44.583238 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fq8q" podUID="110bff76-1581-4379-8468-441a3a81dea2" Nov 03 16:13:44.583717 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:44.583674 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:44.583831 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:44.583802 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7vvq" podUID="8c96f345-2cda-4554-bef3-be416dc50225" Nov 03 16:13:44.583888 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:44.583858 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:44.583962 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:44.583942 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7hw7" podUID="55767581-2059-49bf-8dd0-69e31d5adb83" Nov 03 16:13:44.756094 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:44.755306 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bflh2" event={"ID":"812067ca-eef5-4ab1-a24e-61796ac0721b","Type":"ContainerStarted","Data":"016a572755ad7b5d97976d886cea3383efef9e3dd3578b1193e9c364ce51f402"} Nov 03 16:13:44.796802 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:44.796742 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zmzv6" podStartSLOduration=4.930193158 podStartE2EDuration="8.796720098s" totalImagesPullingTime="3.86652694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:43.786777706 +0000 UTC m=+7.712682593" watchObservedRunningTime="2026-11-03 16:13:44.796720098 +0000 UTC m=+8.722624985" Nov 03 16:13:45.601978 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:45.601927 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zmzv6" Nov 03 16:13:45.602845 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:45.602819 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zmzv6" Nov 03 16:13:45.645286 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:45.644731 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bflh2" podStartSLOduration=4.433502819 podStartE2EDuration="9.644685894s" totalImagesPullingTime="5.211183075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:44.797662135 +0000 UTC m=+8.723567011" watchObservedRunningTime="2026-11-03 16:13:45.644685894 +0000 UTC m=+9.570590960" Nov 03 16:13:45.773172 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:45.772665 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" event={"ID":"a3ce380e-3ddd-42bd-867c-9bba84f2bab4","Type":"ContainerStarted","Data":"b00d3f8ef1488b1e450f8fff79c3aefe380a33f7d9cd28a14b19389ecbc29ad1"} Nov 03 16:13:46.289912 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:46.289874 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:46.290072 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:46.289927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:46.290072 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:46.290063 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:46.290189 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:46.290124 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret podName:8c96f345-2cda-4554-bef3-be416dc50225 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:54.290108209 +0000 UTC m=+18.216013059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret") pod "global-pull-secret-syncer-p7vvq" (UID: "8c96f345-2cda-4554-bef3-be416dc50225") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:46.290453 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:46.290434 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:46.290554 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:46.290475 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs podName:55767581-2059-49bf-8dd0-69e31d5adb83 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:54.290465825 +0000 UTC m=+18.216370676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs") pod "network-metrics-daemon-n7hw7" (UID: "55767581-2059-49bf-8dd0-69e31d5adb83") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:46.390873 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:46.390824 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9slx\" (UniqueName: \"kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx\") pod \"network-check-target-5fq8q\" (UID: \"110bff76-1581-4379-8468-441a3a81dea2\") " pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:46.391058 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:46.391039 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:13:46.391154 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:46.391067 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:13:46.391154 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:46.391080 2577 projected.go:196] Error preparing data for projected volume kube-api-access-l9slx for pod openshift-network-diagnostics/network-check-target-5fq8q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:46.391154 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:46.391145 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx podName:110bff76-1581-4379-8468-441a3a81dea2 nodeName:}" failed. No retries permitted until 2026-11-03 16:13:54.391127654 +0000 UTC m=+18.317032508 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9slx" (UniqueName: "kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx") pod "network-check-target-5fq8q" (UID: "110bff76-1581-4379-8468-441a3a81dea2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:46.409864 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:46.409808 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 16:13:46.579212 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:46.578904 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 16:13:46.579212 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:46.578843 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T16:13:46.409860692Z","UUID":"3a9b608d-f2f5-4544-a1dd-c82fe951ac90","Handler":null,"Name":"","Endpoint":""} Nov 03 16:13:46.579212 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:46.578951 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 16:13:46.581691 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:46.581662 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:46.581816 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:46.581797 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7hw7" podUID="55767581-2059-49bf-8dd0-69e31d5adb83" Nov 03 16:13:46.581952 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:46.581912 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:46.582101 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:46.582021 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fq8q" podUID="110bff76-1581-4379-8468-441a3a81dea2" Nov 03 16:13:46.582167 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:46.582104 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:46.582227 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:46.582178 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7vvq" podUID="8c96f345-2cda-4554-bef3-be416dc50225" Nov 03 16:13:46.779185 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:46.779115 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r6mtg" event={"ID":"1566c753-13b1-4495-9268-05e3676d82bd","Type":"ContainerStarted","Data":"9b15235b362680b2f9963b27065f017a46f3543bafe13086e55060d179de848c"} Nov 03 16:13:46.781769 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:46.781736 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zfshj" event={"ID":"3cbd4c91-69f0-47be-94be-e451d44b4992","Type":"ContainerStarted","Data":"70df56b0a9dcd66cc302588033d30d5d85e91baca0c63824d83a477ba74394f4"} Nov 03 16:13:46.785452 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:46.785411 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" event={"ID":"a3ce380e-3ddd-42bd-867c-9bba84f2bab4","Type":"ContainerStarted","Data":"02f4e124c50104d3b3d638cf50d92cba9e9d1796204fdae8c24185a266364db3"} Nov 03 16:13:46.788035 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:46.788012 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:13:46.788686 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:46.788663 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kmfdd" event={"ID":"7731b228-5421-4121-9228-1f19a850ef9a","Type":"ContainerStarted","Data":"71309164d73b41729af1c44da922bf6ff605a0303bae1b867e5de9ceeea9056b"} Nov 03 16:13:46.926176 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:46.925904 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-kmfdd" podStartSLOduration=3.287520765 podStartE2EDuration="10.925891014s" totalImagesPullingTime="7.638370249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:46.925681512 +0000 UTC m=+10.851586409" watchObservedRunningTime="2026-11-03 16:13:46.925891014 +0000 UTC m=+10.851795887" Nov 03 16:13:46.926350 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:46.926309 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zfshj" podStartSLOduration=3.299697413 podStartE2EDuration="10.926302826s" totalImagesPullingTime="7.626605413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:46.869502805 +0000 UTC m=+10.795407680" watchObservedRunningTime="2026-11-03 16:13:46.926302826 +0000 UTC m=+10.852207700" Nov 03 16:13:47.793234 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:47.793138 2577 generic.go:356] "Generic (PLEG): container finished" podID="1566c753-13b1-4495-9268-05e3676d82bd" containerID="9b15235b362680b2f9963b27065f017a46f3543bafe13086e55060d179de848c" exitCode=0 Nov 03 16:13:47.793758 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:47.793222 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r6mtg" event={"ID":"1566c753-13b1-4495-9268-05e3676d82bd","Type":"ContainerDied","Data":"9b15235b362680b2f9963b27065f017a46f3543bafe13086e55060d179de848c"} Nov 03 16:13:47.796179 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:47.796144 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" event={"ID":"a3ce380e-3ddd-42bd-867c-9bba84f2bab4","Type":"ContainerStarted","Data":"29b3e8efd367e3d8b1357e4d6875203d02113039dc2a3612e681b6cdba3616cd"} Nov 03 16:13:47.885846 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:47.885787 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" podStartSLOduration=3.848339243 podStartE2EDuration="11.88576565s" totalImagesPullingTime="8.037426407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:47.885066324 +0000 UTC m=+11.810971395" watchObservedRunningTime="2026-11-03 16:13:47.88576565 +0000 UTC m=+11.811670526" Nov 03 16:13:48.580655 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:48.580624 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:48.580835 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:48.580783 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7hw7" podUID="55767581-2059-49bf-8dd0-69e31d5adb83" Nov 03 16:13:48.581190 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:48.581169 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:48.581280 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:48.581267 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fq8q" podUID="110bff76-1581-4379-8468-441a3a81dea2" Nov 03 16:13:48.581342 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:48.581333 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:48.581427 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:48.581406 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7vvq" podUID="8c96f345-2cda-4554-bef3-be416dc50225" Nov 03 16:13:48.800598 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:48.800506 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r6mtg" event={"ID":"1566c753-13b1-4495-9268-05e3676d82bd","Type":"ContainerStarted","Data":"40b1d60e5d8628c9955ddc19ac12381b06dd5d4048d3ce8fa90c4c37a2d8d78a"} Nov 03 16:13:48.802308 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:48.802270 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g5w4j" event={"ID":"ec0f2c68-a3bf-4475-ad0d-43346dbd32fd","Type":"ContainerStarted","Data":"13afe5d038459f1313dcc06865f4a7de81ce980de12567174134fa600d8cc185"} Nov 03 16:13:48.887647 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:48.887588 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g5w4j" podStartSLOduration=5.07465125 podStartE2EDuration="12.887569936s" totalImagesPullingTime="7.812918686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:48.887052232 +0000 UTC m=+12.812957105" watchObservedRunningTime="2026-11-03 16:13:48.887569936 +0000 UTC m=+12.813474810" Nov 03 16:13:49.806275 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:49.806235 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" event={"ID":"a7f5be2e-c57e-432f-a5bb-144709f3b492","Type":"ContainerStarted","Data":"86c7b739be48ed1486e6e7b6aec2bd32ea23e6cbcf3b9d1b2bb8fcfeba34b00b"} Nov 03 16:13:49.808932 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:49.808897 2577 generic.go:356] "Generic (PLEG): container finished" podID="1566c753-13b1-4495-9268-05e3676d82bd" containerID="40b1d60e5d8628c9955ddc19ac12381b06dd5d4048d3ce8fa90c4c37a2d8d78a" exitCode=0 Nov 03 16:13:49.809072 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:49.808987 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r6mtg" event={"ID":"1566c753-13b1-4495-9268-05e3676d82bd","Type":"ContainerDied","Data":"40b1d60e5d8628c9955ddc19ac12381b06dd5d4048d3ce8fa90c4c37a2d8d78a"} Nov 03 16:13:49.850191 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:49.850126 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" podStartSLOduration=3.846645603 podStartE2EDuration="13.850112564s" totalImagesPullingTime="10.003466961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:49.849736502 +0000 UTC m=+13.775641372" watchObservedRunningTime="2026-11-03 16:13:49.850112564 +0000 UTC m=+13.776017436" Nov 03 16:13:50.581013 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:50.580974 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:50.581195 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:50.580983 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:50.581195 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:50.581134 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:50.581314 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:50.581111 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7vvq" podUID="8c96f345-2cda-4554-bef3-be416dc50225" Nov 03 16:13:50.581314 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:50.581232 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fq8q" podUID="110bff76-1581-4379-8468-441a3a81dea2" Nov 03 16:13:50.581424 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:50.581347 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7hw7" podUID="55767581-2059-49bf-8dd0-69e31d5adb83" Nov 03 16:13:50.813682 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:50.813645 2577 generic.go:356] "Generic (PLEG): container finished" podID="1566c753-13b1-4495-9268-05e3676d82bd" containerID="0c278f9154d27a09660bd01f4bc4e16bfc60ce47f219597d7b3052f5ba7c6fe7" exitCode=0 Nov 03 16:13:50.814269 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:50.813743 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r6mtg" event={"ID":"1566c753-13b1-4495-9268-05e3676d82bd","Type":"ContainerDied","Data":"0c278f9154d27a09660bd01f4bc4e16bfc60ce47f219597d7b3052f5ba7c6fe7"} Nov 03 16:13:52.580954 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:52.580658 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:52.581480 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:52.581073 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7vvq" podUID="8c96f345-2cda-4554-bef3-be416dc50225" Nov 03 16:13:52.581480 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:52.580725 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:52.581480 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:52.581189 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7hw7" podUID="55767581-2059-49bf-8dd0-69e31d5adb83" Nov 03 16:13:52.581480 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:52.580658 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:52.581480 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:52.581259 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fq8q" podUID="110bff76-1581-4379-8468-441a3a81dea2" Nov 03 16:13:52.822246 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:52.822207 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r6mtg" event={"ID":"1566c753-13b1-4495-9268-05e3676d82bd","Type":"ContainerStarted","Data":"15fb6ad3281402c8396ff124f31ff074812150306809ef8ef6ce26acf4e803f7"} Nov 03 16:13:53.827575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:53.827339 2577 generic.go:356] "Generic (PLEG): container finished" podID="1566c753-13b1-4495-9268-05e3676d82bd" containerID="15fb6ad3281402c8396ff124f31ff074812150306809ef8ef6ce26acf4e803f7" exitCode=0 Nov 03 16:13:53.827575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:53.827416 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r6mtg" event={"ID":"1566c753-13b1-4495-9268-05e3676d82bd","Type":"ContainerDied","Data":"15fb6ad3281402c8396ff124f31ff074812150306809ef8ef6ce26acf4e803f7"} Nov 03 16:13:53.830651 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:53.830625 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" event={"ID":"7ec02fd2-78a5-4813-b6e4-4edefa217fe1","Type":"ContainerStarted","Data":"09f1749ce52d5f13a48520c714b25ffcf963c9045bca2e322d09e33d3882aadc"} Nov 03 16:13:53.830775 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:53.830721 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" event={"ID":"7ec02fd2-78a5-4813-b6e4-4edefa217fe1","Type":"ContainerStarted","Data":"7bd32371585acfc8fb6f68f3603d33464bbea2828c97d8ed44fcaa7c2d88fdca"} Nov 03 16:13:53.830775 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:53.830772 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" event={"ID":"7ec02fd2-78a5-4813-b6e4-4edefa217fe1","Type":"ContainerStarted","Data":"0d20ffee1bd5ccff44b9863934baa13083b4d299d8ea57867d792442f784cbf8"} Nov 03 16:13:53.830894 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:53.830818 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" event={"ID":"7ec02fd2-78a5-4813-b6e4-4edefa217fe1","Type":"ContainerStarted","Data":"a5b591a64239daf58dd913c5eb5318097c1551ae3cf708e8bb10123f75582f3a"} Nov 03 16:13:53.830894 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:53.830863 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" event={"ID":"7ec02fd2-78a5-4813-b6e4-4edefa217fe1","Type":"ContainerStarted","Data":"864362e21b86857465ee40f1f731dbe48133637de06858fc9af276a9d8e80beb"} Nov 03 16:13:53.830982 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:53.830910 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" event={"ID":"7ec02fd2-78a5-4813-b6e4-4edefa217fe1","Type":"ContainerStarted","Data":"7f8be18e169b803aecdd4d2049d20d2abe462b401df95b767b702b0830cba6f8"} Nov 03 16:13:54.354840 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:54.354793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:54.354840 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:54.354844 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:54.355108 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:54.354953 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:54.355108 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:54.354965 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:54.355108 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:54.355007 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret podName:8c96f345-2cda-4554-bef3-be416dc50225 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:10.354995485 +0000 UTC m=+34.280900337 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret") pod "global-pull-secret-syncer-p7vvq" (UID: "8c96f345-2cda-4554-bef3-be416dc50225") : object "kube-system"/"original-pull-secret" not registered Nov 03 16:13:54.355108 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:54.355028 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs podName:55767581-2059-49bf-8dd0-69e31d5adb83 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:10.355013327 +0000 UTC m=+34.280918189 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs") pod "network-metrics-daemon-n7hw7" (UID: "55767581-2059-49bf-8dd0-69e31d5adb83") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 16:13:54.456100 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:54.456062 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9slx\" (UniqueName: \"kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx\") pod \"network-check-target-5fq8q\" (UID: \"110bff76-1581-4379-8468-441a3a81dea2\") " pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:54.456280 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:54.456209 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 16:13:54.456280 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:54.456226 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 16:13:54.456280 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:54.456235 2577 projected.go:196] Error preparing data for projected volume kube-api-access-l9slx for pod openshift-network-diagnostics/network-check-target-5fq8q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:54.456280 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:54.456281 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx podName:110bff76-1581-4379-8468-441a3a81dea2 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:10.45626827 +0000 UTC m=+34.382173121 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l9slx" (UniqueName: "kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx") pod "network-check-target-5fq8q" (UID: "110bff76-1581-4379-8468-441a3a81dea2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 16:13:54.580541 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:54.580508 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:54.580541 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:54.580521 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:54.580787 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:54.580608 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:54.580787 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:54.580616 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7hw7" podUID="55767581-2059-49bf-8dd0-69e31d5adb83" Nov 03 16:13:54.580787 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:54.580684 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fq8q" podUID="110bff76-1581-4379-8468-441a3a81dea2" Nov 03 16:13:54.580905 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:54.580856 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7vvq" podUID="8c96f345-2cda-4554-bef3-be416dc50225" Nov 03 16:13:54.836241 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:54.836210 2577 generic.go:356] "Generic (PLEG): container finished" podID="1566c753-13b1-4495-9268-05e3676d82bd" containerID="a23d50a3132cdeee005ca133ad5efbc08d4c4f76a7bd9a750c694db889baf0e3" exitCode=0 Nov 03 16:13:54.836754 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:54.836263 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r6mtg" event={"ID":"1566c753-13b1-4495-9268-05e3676d82bd","Type":"ContainerDied","Data":"a23d50a3132cdeee005ca133ad5efbc08d4c4f76a7bd9a750c694db889baf0e3"} Nov 03 16:13:55.842238 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:55.842200 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r6mtg" event={"ID":"1566c753-13b1-4495-9268-05e3676d82bd","Type":"ContainerStarted","Data":"7091987990cfd58c2319a730275cb2a1281fb1e2997b6ae75a19a5de9a5d7c45"} Nov 03 16:13:55.845283 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:55.845258 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" event={"ID":"7ec02fd2-78a5-4813-b6e4-4edefa217fe1","Type":"ContainerStarted","Data":"f911730aede0f6087d15387a5ec8cc3814e416f34739464428c4a1b7b195f97c"} Nov 03 16:13:55.917900 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:55.917790 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r6mtg" podStartSLOduration=9.563261679 podStartE2EDuration="19.917774079s" totalImagesPullingTime="10.3545124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:55.902894837 +0000 UTC m=+19.828799710" watchObservedRunningTime="2026-11-03 16:13:55.917774079 +0000 UTC m=+19.843678951" Nov 03 16:13:56.582245 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:56.582208 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:56.582497 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:56.582316 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:56.582497 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:56.582350 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:56.582497 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:56.582362 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7hw7" podUID="55767581-2059-49bf-8dd0-69e31d5adb83" Nov 03 16:13:56.582497 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:56.582425 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fq8q" podUID="110bff76-1581-4379-8468-441a3a81dea2" Nov 03 16:13:56.582742 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:56.582488 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7vvq" podUID="8c96f345-2cda-4554-bef3-be416dc50225" Nov 03 16:13:57.853317 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:57.853283 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" event={"ID":"7ec02fd2-78a5-4813-b6e4-4edefa217fe1","Type":"ContainerStarted","Data":"ed7fb1abcc8e630e737dd50678ab34235f6a5b2343a6c1f97b54a1793d7034ab"} Nov 03 16:13:57.854002 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:57.853662 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:57.868593 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:57.868571 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:57.949492 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:57.949304 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" podStartSLOduration=6.596073734 podStartE2EDuration="20.949291061s" totalImagesPullingTime="14.353217327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 16:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:13:57.948756746 +0000 UTC m=+21.874661623" watchObservedRunningTime="2026-11-03 16:13:57.949291061 +0000 UTC m=+21.875195933" Nov 03 16:13:58.580505 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:58.580468 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:58.580505 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:58.580494 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:58.580505 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:58.580486 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:58.580765 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:58.580602 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fq8q" podUID="110bff76-1581-4379-8468-441a3a81dea2" Nov 03 16:13:58.580765 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:58.580656 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7hw7" podUID="55767581-2059-49bf-8dd0-69e31d5adb83" Nov 03 16:13:58.580849 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:58.580757 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7vvq" podUID="8c96f345-2cda-4554-bef3-be416dc50225" Nov 03 16:13:58.855870 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:58.855796 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:58.855870 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:58.855866 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:58.871460 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:58.871439 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:13:59.850759 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:59.850727 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5fq8q"] Nov 03 16:13:59.850942 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:59.850903 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:13:59.851039 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:59.851020 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fq8q" podUID="110bff76-1581-4379-8468-441a3a81dea2" Nov 03 16:13:59.851247 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:59.851222 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n7hw7"] Nov 03 16:13:59.851356 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:59.851343 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:13:59.851478 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:59.851458 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7hw7" podUID="55767581-2059-49bf-8dd0-69e31d5adb83" Nov 03 16:13:59.851817 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:59.851802 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p7vvq"] Nov 03 16:13:59.851884 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:13:59.851875 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:13:59.851952 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:13:59.851938 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7vvq" podUID="8c96f345-2cda-4554-bef3-be416dc50225" Nov 03 16:14:01.570016 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:01.569979 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zmzv6" Nov 03 16:14:01.570519 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:01.570110 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:14:01.570590 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:01.570572 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zmzv6" Nov 03 16:14:01.581078 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:01.581055 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:14:01.581193 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:01.581056 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:14:01.581193 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:01.581145 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7vvq" podUID="8c96f345-2cda-4554-bef3-be416dc50225" Nov 03 16:14:01.581252 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:01.581056 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:14:01.581252 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:01.581230 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7hw7" podUID="55767581-2059-49bf-8dd0-69e31d5adb83" Nov 03 16:14:01.581330 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:01.581312 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fq8q" podUID="110bff76-1581-4379-8468-441a3a81dea2" Nov 03 16:14:03.580743 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:03.580687 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:14:03.580743 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:03.580716 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:14:03.581342 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:03.580846 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n7hw7" podUID="55767581-2059-49bf-8dd0-69e31d5adb83" Nov 03 16:14:03.581342 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:03.580891 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:14:03.581342 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:03.580986 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p7vvq" podUID="8c96f345-2cda-4554-bef3-be416dc50225" Nov 03 16:14:03.581342 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:03.581052 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5fq8q" podUID="110bff76-1581-4379-8468-441a3a81dea2" Nov 03 16:14:05.343051 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.343023 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeReady" Nov 03 16:14:05.343439 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.343143 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 16:14:05.397988 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.397952 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j8t6k"] Nov 03 16:14:05.400846 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.400827 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4j5k5"] Nov 03 16:14:05.400977 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.400953 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-j8t6k" Nov 03 16:14:05.403690 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.403660 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4j5k5" Nov 03 16:14:05.404023 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.403961 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hxr8b"] Nov 03 16:14:05.406822 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.406798 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68c8fc97c8-vxzz9"] Nov 03 16:14:05.406955 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.406908 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hxr8b" Nov 03 16:14:05.409637 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.409617 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-r755z"] Nov 03 16:14:05.409761 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.409744 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.412370 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.412349 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt"] Nov 03 16:14:05.412467 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.412456 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:05.416864 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.416845 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5dc56589f5-wsj6w"] Nov 03 16:14:05.417282 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.417266 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:05.419553 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.419532 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-964q9"] Nov 03 16:14:05.419672 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.419656 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:05.426018 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.425996 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-88dnt"] Nov 03 16:14:05.426211 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.426193 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.426669 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.426650 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 16:14:05.426831 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.426670 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 16:14:05.429732 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.429211 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d945bd975-wtl6x"] Nov 03 16:14:05.429983 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.429967 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-88dnt" Nov 03 16:14:05.431949 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.431928 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-tnh8w"] Nov 03 16:14:05.432062 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.432047 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.434616 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.434359 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 16:14:05.434616 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.434583 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 16:14:05.434803 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.434674 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 16:14:05.434803 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.434757 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-d6vzh\"" Nov 03 16:14:05.434913 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.434899 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 16:14:05.435012 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.434995 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qj6r5\"" Nov 03 16:14:05.435124 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.435089 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4"] Nov 03 16:14:05.435233 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.435216 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:05.438272 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.437975 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft"] Nov 03 16:14:05.438272 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.438165 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 16:14:05.438542 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.438520 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" Nov 03 16:14:05.441669 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.441601 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4"] Nov 03 16:14:05.441827 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.441716 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" Nov 03 16:14:05.446760 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.446738 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cm2k2"] Nov 03 16:14:05.447850 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.446943 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:05.453011 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.452988 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jdthd"] Nov 03 16:14:05.453276 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.453125 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm2k2" Nov 03 16:14:05.457017 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.456997 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xfk72"] Nov 03 16:14:05.457488 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.457467 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jdthd" Nov 03 16:14:05.460348 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.460330 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j8t6k"] Nov 03 16:14:05.460462 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.460366 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4j5k5"] Nov 03 16:14:05.460462 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.460390 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt"] Nov 03 16:14:05.460462 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.460413 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68c8fc97c8-vxzz9"] Nov 03 16:14:05.460462 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.460437 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-88dnt"] Nov 03 16:14:05.460462 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.460463 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5dc56589f5-wsj6w"] Nov 03 16:14:05.460723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.460480 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-964q9"] Nov 03 16:14:05.460723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.460478 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:05.460723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.460492 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cm2k2"] Nov 03 16:14:05.460723 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.460680 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-tnh8w"] Nov 03 16:14:05.460879 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.460741 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d945bd975-wtl6x"] Nov 03 16:14:05.460879 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.460766 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4"] Nov 03 16:14:05.460879 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.460799 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hxr8b"] Nov 03 16:14:05.472172 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.472144 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-r755z"] Nov 03 16:14:05.480339 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.480314 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4"] Nov 03 16:14:05.481108 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.481085 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft"] Nov 03 16:14:05.487994 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.487970 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jdthd"] Nov 03 16:14:05.488101 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.488020 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xfk72"] Nov 03 16:14:05.526728 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.526660 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 16:14:05.526852 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.526842 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 16:14:05.526957 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.526875 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 16:14:05.526957 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.526887 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 16:14:05.526957 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.526887 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 16:14:05.526957 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.526875 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:14:05.527234 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527032 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 16:14:05.527234 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527063 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 16:14:05.527234 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527168 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 16:14:05.527234 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527212 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 16:14:05.527586 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527298 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:14:05.527586 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527341 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 16:14:05.527586 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527355 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 16:14:05.527586 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527398 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 16:14:05.527586 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527420 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:14:05.527586 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527444 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 16:14:05.527586 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527458 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 16:14:05.527586 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527475 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:14:05.527586 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527560 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 16:14:05.527586 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527585 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 16:14:05.528186 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527624 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 16:14:05.528186 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527656 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 16:14:05.528186 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527667 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 16:14:05.528186 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527725 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 16:14:05.528186 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527741 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 16:14:05.528186 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527741 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 16:14:05.528186 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527773 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:14:05.528186 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.527788 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 16:14:05.529027 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529006 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:05.529115 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:05.529115 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529048 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1580792e-eb86-4d79-92a7-8055e6a8aaff-config\") pod \"service-ca-operator-544d749c8c-qqcft\" (UID: \"1580792e-eb86-4d79-92a7-8055e6a8aaff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" Nov 03 16:14:05.529115 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529064 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-snapshots\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.529115 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529080 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-trusted-ca\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.529312 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529165 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl4vw\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-kube-api-access-rl4vw\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.529312 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529195 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-image-registry-private-configuration\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.529312 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529217 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:05.529312 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529259 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-service-ca-bundle\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.529312 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529284 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-stats-auth\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:05.529312 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529307 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-trusted-ca\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.529612 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529324 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1580792e-eb86-4d79-92a7-8055e6a8aaff-serving-cert\") pod \"service-ca-operator-544d749c8c-qqcft\" (UID: \"1580792e-eb86-4d79-92a7-8055e6a8aaff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" Nov 03 16:14:05.529612 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529350 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.529612 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529394 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.529612 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529410 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g2jb\" (UniqueName: \"kubernetes.io/projected/78e9f545-8400-48b6-968d-eb65dc94b471-kube-api-access-9g2jb\") pod \"kas-connection-checker-685895c7c9-j8t6k\" (UID: \"78e9f545-8400-48b6-968d-eb65dc94b471\") " pod="kube-system/kas-connection-checker-685895c7c9-j8t6k" Nov 03 16:14:05.529612 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529426 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42fxm\" (UniqueName: \"kubernetes.io/projected/4cbc83f2-64b1-4804-a180-43df6459ffa4-kube-api-access-42fxm\") pod \"kas-connection-checker-685895c7c9-4j5k5\" (UID: \"4cbc83f2-64b1-4804-a180-43df6459ffa4\") " pod="kube-system/kas-connection-checker-685895c7c9-4j5k5" Nov 03 16:14:05.529612 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529441 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75xq7\" (UniqueName: \"kubernetes.io/projected/66b43213-9ec2-4164-9f8d-d7621bd57eb3-kube-api-access-75xq7\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:05.529612 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529465 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.529612 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529509 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4dbx\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-kube-api-access-z4dbx\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.529612 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529544 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fx4n\" (UniqueName: \"kubernetes.io/projected/1580792e-eb86-4d79-92a7-8055e6a8aaff-kube-api-access-8fx4n\") pod \"service-ca-operator-544d749c8c-qqcft\" (UID: \"1580792e-eb86-4d79-92a7-8055e6a8aaff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" Nov 03 16:14:05.529612 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529585 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tszlg\" (UniqueName: \"kubernetes.io/projected/ba6c03a7-1ba9-48c0-b67b-6fb26263425e-kube-api-access-tszlg\") pod \"network-check-source-6f78d59f99-cm2k2\" (UID: \"ba6c03a7-1ba9-48c0-b67b-6fb26263425e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm2k2" Nov 03 16:14:05.529612 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529601 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-ca-trust-extracted\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.530143 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529621 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r755z\" (UID: \"2c812633-b15d-4a30-982d-80d276ec78c7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:05.530143 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529642 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2c812633-b15d-4a30-982d-80d276ec78c7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-r755z\" (UID: \"2c812633-b15d-4a30-982d-80d276ec78c7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:05.530143 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529656 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:05.530143 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529671 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-serving-cert\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.530143 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529710 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03cc593d-32bc-4453-8a45-2b546f862acf-ca-trust-extracted\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.530143 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529749 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-przlh\" (UniqueName: \"kubernetes.io/projected/5ced7a93-d9f0-44de-9a76-34a84e41843b-kube-api-access-przlh\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:05.530143 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529783 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtbz7\" (UniqueName: \"kubernetes.io/projected/bf8c316f-51ce-4c69-865f-1d481ff30630-kube-api-access-gtbz7\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4\" (UID: \"bf8c316f-51ce-4c69-865f-1d481ff30630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" Nov 03 16:14:05.530143 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-default-certificate\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:05.530143 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529853 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-registry-certificates\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.530143 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529896 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzhxd\" (UniqueName: \"kubernetes.io/projected/4b370319-6de8-41e2-ae5f-21267a2fba74-kube-api-access-pzhxd\") pod \"kas-connection-checker-685895c7c9-hxr8b\" (UID: \"4b370319-6de8-41e2-ae5f-21267a2fba74\") " pod="kube-system/kas-connection-checker-685895c7c9-hxr8b" Nov 03 16:14:05.530143 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529929 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdzb4\" (UniqueName: \"kubernetes.io/projected/8f85dfa8-9158-437d-b151-d8eb8fe65b7b-kube-api-access-xdzb4\") pod \"ingress-canary-jdthd\" (UID: \"8f85dfa8-9158-437d-b151-d8eb8fe65b7b\") " pod="openshift-ingress-canary/ingress-canary-jdthd" Nov 03 16:14:05.530143 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529950 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/216c0d66-a3ec-4748-b761-3420536bfa3a-serving-cert\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:05.530143 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.529998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-installation-pull-secrets\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.530143 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rznq\" (UniqueName: \"kubernetes.io/projected/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-kube-api-access-4rznq\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.530143 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530051 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-bound-sa-token\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.530877 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530091 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:05.530877 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530109 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phmbx\" (UniqueName: \"kubernetes.io/projected/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-kube-api-access-phmbx\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:05.530877 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530133 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f85dfa8-9158-437d-b151-d8eb8fe65b7b-cert\") pod \"ingress-canary-jdthd\" (UID: \"8f85dfa8-9158-437d-b151-d8eb8fe65b7b\") " pod="openshift-ingress-canary/ingress-canary-jdthd" Nov 03 16:14:05.530877 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530161 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-image-registry-private-configuration\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.530877 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530180 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svnps\" (UniqueName: \"kubernetes.io/projected/c2ded168-1814-4922-9eee-00caf304b8af-kube-api-access-svnps\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:05.530877 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530195 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ced7a93-d9f0-44de-9a76-34a84e41843b-tmp-dir\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:05.530877 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530212 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-certificates\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.530877 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530241 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vdm\" (UniqueName: \"kubernetes.io/projected/fa01a744-9121-4f6a-b280-dba215c1ab8f-kube-api-access-d4vdm\") pod \"volume-data-source-validator-58c9f5f46d-88dnt\" (UID: \"fa01a744-9121-4f6a-b280-dba215c1ab8f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-88dnt" Nov 03 16:14:05.530877 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530269 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf8c316f-51ce-4c69-865f-1d481ff30630-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4\" (UID: \"bf8c316f-51ce-4c69-865f-1d481ff30630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" Nov 03 16:14:05.530877 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:05.530877 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530309 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:05.530877 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530326 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz7ht\" (UniqueName: \"kubernetes.io/projected/216c0d66-a3ec-4748-b761-3420536bfa3a-kube-api-access-wz7ht\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:05.530877 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530365 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf8c316f-51ce-4c69-865f-1d481ff30630-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4\" (UID: \"bf8c316f-51ce-4c69-865f-1d481ff30630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" Nov 03 16:14:05.530877 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530388 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-bound-sa-token\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.531546 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530405 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/216c0d66-a3ec-4748-b761-3420536bfa3a-trusted-ca\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:05.531546 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530451 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-installation-pull-secrets\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.531546 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530468 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5ced7a93-d9f0-44de-9a76-34a84e41843b-config-volume\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:05.531546 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530500 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/216c0d66-a3ec-4748-b761-3420536bfa3a-config\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:05.531546 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.530516 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-tmp\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.542608 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.542590 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 16:14:05.542608 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.542600 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 16:14:05.542608 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.542593 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 16:14:05.542812 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.542609 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 16:14:05.542812 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.542633 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 16:14:05.542812 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.542634 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 16:14:05.542812 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.542652 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 16:14:05.542812 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.542679 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 16:14:05.542812 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.542688 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 16:14:05.542812 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.542731 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 16:14:05.542812 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.542774 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s5c6x\"" Nov 03 16:14:05.543142 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.542819 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nnkq7\"" Nov 03 16:14:05.543142 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.542857 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 16:14:05.557678 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.557653 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 16:14:05.557978 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.557957 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 16:14:05.558658 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.558633 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 16:14:05.581332 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.581313 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:14:05.581332 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.581324 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:14:05.581473 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.581443 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:14:05.631377 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.631354 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-image-registry-private-configuration\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.631511 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.631386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:05.631511 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.631404 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-service-ca-bundle\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.631511 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.631421 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-stats-auth\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:05.631511 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.631472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-trusted-ca\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.631511 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.631497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1580792e-eb86-4d79-92a7-8055e6a8aaff-serving-cert\") pod \"service-ca-operator-544d749c8c-qqcft\" (UID: \"1580792e-eb86-4d79-92a7-8055e6a8aaff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" Nov 03 16:14:05.631786 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:05.631518 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 16:14:05.631786 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:05.631578 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls podName:c2ded168-1814-4922-9eee-00caf304b8af nodeName:}" failed. No retries permitted until 2026-11-03 16:14:06.13155695 +0000 UTC m=+30.057461804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vmcs4" (UID: "c2ded168-1814-4922-9eee-00caf304b8af") : secret "samples-operator-tls" not found Nov 03 16:14:05.631901 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.631519 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.631901 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.631890 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.631993 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.631918 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g2jb\" (UniqueName: \"kubernetes.io/projected/78e9f545-8400-48b6-968d-eb65dc94b471-kube-api-access-9g2jb\") pod \"kas-connection-checker-685895c7c9-j8t6k\" (UID: \"78e9f545-8400-48b6-968d-eb65dc94b471\") " pod="kube-system/kas-connection-checker-685895c7c9-j8t6k" Nov 03 16:14:05.631993 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.631943 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42fxm\" (UniqueName: \"kubernetes.io/projected/4cbc83f2-64b1-4804-a180-43df6459ffa4-kube-api-access-42fxm\") pod \"kas-connection-checker-685895c7c9-4j5k5\" (UID: \"4cbc83f2-64b1-4804-a180-43df6459ffa4\") " pod="kube-system/kas-connection-checker-685895c7c9-4j5k5" Nov 03 16:14:05.631993 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.631971 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xq7\" (UniqueName: \"kubernetes.io/projected/66b43213-9ec2-4164-9f8d-d7621bd57eb3-kube-api-access-75xq7\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:05.632135 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.631994 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.632135 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.632030 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4dbx\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-kube-api-access-z4dbx\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.632135 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.632055 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fx4n\" (UniqueName: \"kubernetes.io/projected/1580792e-eb86-4d79-92a7-8055e6a8aaff-kube-api-access-8fx4n\") pod \"service-ca-operator-544d749c8c-qqcft\" (UID: \"1580792e-eb86-4d79-92a7-8055e6a8aaff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" Nov 03 16:14:05.632135 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.632085 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tszlg\" (UniqueName: \"kubernetes.io/projected/ba6c03a7-1ba9-48c0-b67b-6fb26263425e-kube-api-access-tszlg\") pod \"network-check-source-6f78d59f99-cm2k2\" (UID: \"ba6c03a7-1ba9-48c0-b67b-6fb26263425e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm2k2" Nov 03 16:14:05.632135 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.632109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-ca-trust-extracted\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.632384 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.632136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r755z\" (UID: \"2c812633-b15d-4a30-982d-80d276ec78c7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:05.632384 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.632161 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2c812633-b15d-4a30-982d-80d276ec78c7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-r755z\" (UID: \"2c812633-b15d-4a30-982d-80d276ec78c7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:05.632384 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.632186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:05.632384 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.632209 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-serving-cert\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.632384 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.632239 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03cc593d-32bc-4453-8a45-2b546f862acf-ca-trust-extracted\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.632384 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.632265 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-przlh\" (UniqueName: \"kubernetes.io/projected/5ced7a93-d9f0-44de-9a76-34a84e41843b-kube-api-access-przlh\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:05.632663 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.632492 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.632663 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.632563 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-ca-trust-extracted\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.632663 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:05.632566 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:05.632663 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:05.632590 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d945bd975-wtl6x: secret "image-registry-tls" not found Nov 03 16:14:05.632663 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:05.632633 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls podName:03cc593d-32bc-4453-8a45-2b546f862acf nodeName:}" failed. No retries permitted until 2026-11-03 16:14:06.132618304 +0000 UTC m=+30.058523171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls") pod "image-registry-7d945bd975-wtl6x" (UID: "03cc593d-32bc-4453-8a45-2b546f862acf") : secret "image-registry-tls" not found Nov 03 16:14:05.632938 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:05.632722 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 16:14:05.632938 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:05.632760 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert podName:2c812633-b15d-4a30-982d-80d276ec78c7 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:06.132748621 +0000 UTC m=+30.058653488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r755z" (UID: "2c812633-b15d-4a30-982d-80d276ec78c7") : secret "networking-console-plugin-cert" not found Nov 03 16:14:05.632938 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:05.632801 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:05.632938 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:05.632818 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68c8fc97c8-vxzz9: secret "image-registry-tls" not found Nov 03 16:14:05.632938 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:05.632874 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls podName:a13e4eb2-b4f2-4f66-85d3-8bedb078823c nodeName:}" failed. No retries permitted until 2026-11-03 16:14:06.132858783 +0000 UTC m=+30.058763655 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls") pod "image-registry-68c8fc97c8-vxzz9" (UID: "a13e4eb2-b4f2-4f66-85d3-8bedb078823c") : secret "image-registry-tls" not found Nov 03 16:14:05.633185 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633132 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtbz7\" (UniqueName: \"kubernetes.io/projected/bf8c316f-51ce-4c69-865f-1d481ff30630-kube-api-access-gtbz7\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4\" (UID: \"bf8c316f-51ce-4c69-865f-1d481ff30630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" Nov 03 16:14:05.633185 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633168 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-default-certificate\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:05.633290 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633204 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-registry-certificates\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.633290 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzhxd\" (UniqueName: \"kubernetes.io/projected/4b370319-6de8-41e2-ae5f-21267a2fba74-kube-api-access-pzhxd\") pod \"kas-connection-checker-685895c7c9-hxr8b\" (UID: \"4b370319-6de8-41e2-ae5f-21267a2fba74\") " pod="kube-system/kas-connection-checker-685895c7c9-hxr8b" Nov 03 16:14:05.633290 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633264 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzb4\" (UniqueName: \"kubernetes.io/projected/8f85dfa8-9158-437d-b151-d8eb8fe65b7b-kube-api-access-xdzb4\") pod \"ingress-canary-jdthd\" (UID: \"8f85dfa8-9158-437d-b151-d8eb8fe65b7b\") " pod="openshift-ingress-canary/ingress-canary-jdthd" Nov 03 16:14:05.633434 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633290 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/216c0d66-a3ec-4748-b761-3420536bfa3a-serving-cert\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:05.633434 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633322 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-installation-pull-secrets\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.633434 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633350 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rznq\" (UniqueName: \"kubernetes.io/projected/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-kube-api-access-4rznq\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.633434 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633359 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 16:14:05.633434 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633375 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-bound-sa-token\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.633434 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633391 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2c812633-b15d-4a30-982d-80d276ec78c7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-r755z\" (UID: \"2c812633-b15d-4a30-982d-80d276ec78c7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:05.633434 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633408 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7jhdd\"" Nov 03 16:14:05.633434 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633408 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:05.633815 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633439 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phmbx\" (UniqueName: \"kubernetes.io/projected/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-kube-api-access-phmbx\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:05.633815 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633466 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 16:14:05.633815 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:05.633478 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 16:14:05.633815 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.632210 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-service-ca-bundle\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.633815 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:05.633520 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs podName:66b43213-9ec2-4164-9f8d-d7621bd57eb3 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:06.13350568 +0000 UTC m=+30.059410548 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs") pod "router-default-5dc56589f5-wsj6w" (UID: "66b43213-9ec2-4164-9f8d-d7621bd57eb3") : secret "router-metrics-certs-default" not found Nov 03 16:14:05.633815 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633573 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6cp7r\"" Nov 03 16:14:05.633815 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.632562 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-trusted-ca\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.633815 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f85dfa8-9158-437d-b151-d8eb8fe65b7b-cert\") pod \"ingress-canary-jdthd\" (UID: \"8f85dfa8-9158-437d-b151-d8eb8fe65b7b\") " pod="openshift-ingress-canary/ingress-canary-jdthd" Nov 03 16:14:05.633815 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633635 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-image-registry-private-configuration\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.633815 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633661 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svnps\" (UniqueName: \"kubernetes.io/projected/c2ded168-1814-4922-9eee-00caf304b8af-kube-api-access-svnps\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:05.633815 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633688 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ced7a93-d9f0-44de-9a76-34a84e41843b-tmp-dir\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:05.633815 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633740 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-certificates\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.633815 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633766 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vdm\" (UniqueName: \"kubernetes.io/projected/fa01a744-9121-4f6a-b280-dba215c1ab8f-kube-api-access-d4vdm\") pod \"volume-data-source-validator-58c9f5f46d-88dnt\" (UID: \"fa01a744-9121-4f6a-b280-dba215c1ab8f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-88dnt" Nov 03 16:14:05.633815 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633800 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf8c316f-51ce-4c69-865f-1d481ff30630-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4\" (UID: \"bf8c316f-51ce-4c69-865f-1d481ff30630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" Nov 03 16:14:05.633815 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633830 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:05.634729 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633857 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:05.634729 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633863 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03cc593d-32bc-4453-8a45-2b546f862acf-ca-trust-extracted\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.634729 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633899 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz7ht\" (UniqueName: \"kubernetes.io/projected/216c0d66-a3ec-4748-b761-3420536bfa3a-kube-api-access-wz7ht\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:05.634729 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf8c316f-51ce-4c69-865f-1d481ff30630-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4\" (UID: \"bf8c316f-51ce-4c69-865f-1d481ff30630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" Nov 03 16:14:05.634729 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633968 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-bound-sa-token\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.634729 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633991 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/216c0d66-a3ec-4748-b761-3420536bfa3a-trusted-ca\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:05.634729 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.634038 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-installation-pull-secrets\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.634729 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.634067 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5ced7a93-d9f0-44de-9a76-34a84e41843b-config-volume\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:05.634729 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.634097 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/216c0d66-a3ec-4748-b761-3420536bfa3a-config\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:05.634729 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.634121 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-tmp\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.634729 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.634166 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:05.634729 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.634190 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:05.634729 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.634212 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1580792e-eb86-4d79-92a7-8055e6a8aaff-config\") pod \"service-ca-operator-544d749c8c-qqcft\" (UID: \"1580792e-eb86-4d79-92a7-8055e6a8aaff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" Nov 03 16:14:05.634729 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.634233 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-snapshots\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.634729 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.634254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-trusted-ca\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.634729 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.634275 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl4vw\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-kube-api-access-rl4vw\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.635426 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.634564 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:05.635426 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.635002 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ced7a93-d9f0-44de-9a76-34a84e41843b-tmp-dir\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:05.635426 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.635022 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/216c0d66-a3ec-4748-b761-3420536bfa3a-trusted-ca\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:05.635562 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.635502 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf8c316f-51ce-4c69-865f-1d481ff30630-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4\" (UID: \"bf8c316f-51ce-4c69-865f-1d481ff30630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" Nov 03 16:14:05.635728 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.635632 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-certificates\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.638154 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:05.635844 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle podName:66b43213-9ec2-4164-9f8d-d7621bd57eb3 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:06.135828688 +0000 UTC m=+30.061733562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle") pod "router-default-5dc56589f5-wsj6w" (UID: "66b43213-9ec2-4164-9f8d-d7621bd57eb3") : configmap references non-existent config key: service-ca.crt Nov 03 16:14:05.638154 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.636271 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-registry-certificates\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.638154 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.636304 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5ced7a93-d9f0-44de-9a76-34a84e41843b-config-volume\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:05.638154 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:05.636636 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 16:14:05.638154 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:05.636760 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls podName:5ced7a93-d9f0-44de-9a76-34a84e41843b nodeName:}" failed. No retries permitted until 2026-11-03 16:14:06.136742643 +0000 UTC m=+30.062647511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls") pod "dns-default-xfk72" (UID: "5ced7a93-d9f0-44de-9a76-34a84e41843b") : secret "dns-default-metrics-tls" not found Nov 03 16:14:05.638154 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.636882 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/216c0d66-a3ec-4748-b761-3420536bfa3a-config\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:05.638154 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.636987 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1580792e-eb86-4d79-92a7-8055e6a8aaff-config\") pod \"service-ca-operator-544d749c8c-qqcft\" (UID: \"1580792e-eb86-4d79-92a7-8055e6a8aaff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" Nov 03 16:14:05.638154 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.637344 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-tmp\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.638154 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.637376 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-serving-cert\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.638154 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.637439 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-installation-pull-secrets\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.638154 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.637494 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-stats-auth\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:05.638154 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.633291 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-67www\"" Nov 03 16:14:05.638154 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.637741 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-image-registry-private-configuration\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.638154 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.637913 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-snapshots\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.638154 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:05.637956 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:05.638154 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:05.638003 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls podName:e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:06.13798819 +0000 UTC m=+30.063893068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bw6bt" (UID: "e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28") : secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:05.639078 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.638332 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-trusted-ca\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.639078 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.638393 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:05.639189 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.639102 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf8c316f-51ce-4c69-865f-1d481ff30630-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4\" (UID: \"bf8c316f-51ce-4c69-865f-1d481ff30630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" Nov 03 16:14:05.639545 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.639522 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-default-certificate\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:05.639611 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.639563 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-image-registry-private-configuration\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.639793 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.639778 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1580792e-eb86-4d79-92a7-8055e6a8aaff-serving-cert\") pod \"service-ca-operator-544d749c8c-qqcft\" (UID: \"1580792e-eb86-4d79-92a7-8055e6a8aaff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" Nov 03 16:14:05.639871 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.639850 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f85dfa8-9158-437d-b151-d8eb8fe65b7b-cert\") pod \"ingress-canary-jdthd\" (UID: \"8f85dfa8-9158-437d-b151-d8eb8fe65b7b\") " pod="openshift-ingress-canary/ingress-canary-jdthd" Nov 03 16:14:05.640384 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.640362 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/216c0d66-a3ec-4748-b761-3420536bfa3a-serving-cert\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:05.641067 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.641049 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-installation-pull-secrets\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.659993 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.659969 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g2jb\" (UniqueName: \"kubernetes.io/projected/78e9f545-8400-48b6-968d-eb65dc94b471-kube-api-access-9g2jb\") pod \"kas-connection-checker-685895c7c9-j8t6k\" (UID: \"78e9f545-8400-48b6-968d-eb65dc94b471\") " pod="kube-system/kas-connection-checker-685895c7c9-j8t6k" Nov 03 16:14:05.660446 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.660421 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fx4n\" (UniqueName: \"kubernetes.io/projected/1580792e-eb86-4d79-92a7-8055e6a8aaff-kube-api-access-8fx4n\") pod \"service-ca-operator-544d749c8c-qqcft\" (UID: \"1580792e-eb86-4d79-92a7-8055e6a8aaff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" Nov 03 16:14:05.662150 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.661837 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl4vw\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-kube-api-access-rl4vw\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.662241 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.662180 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzhxd\" (UniqueName: \"kubernetes.io/projected/4b370319-6de8-41e2-ae5f-21267a2fba74-kube-api-access-pzhxd\") pod \"kas-connection-checker-685895c7c9-hxr8b\" (UID: \"4b370319-6de8-41e2-ae5f-21267a2fba74\") " pod="kube-system/kas-connection-checker-685895c7c9-hxr8b" Nov 03 16:14:05.662303 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.661818 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tszlg\" (UniqueName: \"kubernetes.io/projected/ba6c03a7-1ba9-48c0-b67b-6fb26263425e-kube-api-access-tszlg\") pod \"network-check-source-6f78d59f99-cm2k2\" (UID: \"ba6c03a7-1ba9-48c0-b67b-6fb26263425e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm2k2" Nov 03 16:14:05.663040 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.663016 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42fxm\" (UniqueName: \"kubernetes.io/projected/4cbc83f2-64b1-4804-a180-43df6459ffa4-kube-api-access-42fxm\") pod \"kas-connection-checker-685895c7c9-4j5k5\" (UID: \"4cbc83f2-64b1-4804-a180-43df6459ffa4\") " pod="kube-system/kas-connection-checker-685895c7c9-4j5k5" Nov 03 16:14:05.664223 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.664180 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4dbx\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-kube-api-access-z4dbx\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.664948 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.664926 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-przlh\" (UniqueName: \"kubernetes.io/projected/5ced7a93-d9f0-44de-9a76-34a84e41843b-kube-api-access-przlh\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:05.667302 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.667255 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdzb4\" (UniqueName: \"kubernetes.io/projected/8f85dfa8-9158-437d-b151-d8eb8fe65b7b-kube-api-access-xdzb4\") pod \"ingress-canary-jdthd\" (UID: \"8f85dfa8-9158-437d-b151-d8eb8fe65b7b\") " pod="openshift-ingress-canary/ingress-canary-jdthd" Nov 03 16:14:05.667649 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.667628 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svnps\" (UniqueName: \"kubernetes.io/projected/c2ded168-1814-4922-9eee-00caf304b8af-kube-api-access-svnps\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:05.667970 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.667951 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz7ht\" (UniqueName: \"kubernetes.io/projected/216c0d66-a3ec-4748-b761-3420536bfa3a-kube-api-access-wz7ht\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:05.668675 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.668654 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75xq7\" (UniqueName: \"kubernetes.io/projected/66b43213-9ec2-4164-9f8d-d7621bd57eb3-kube-api-access-75xq7\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:05.668797 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.668731 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-bound-sa-token\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:05.668894 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.668871 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtbz7\" (UniqueName: \"kubernetes.io/projected/bf8c316f-51ce-4c69-865f-1d481ff30630-kube-api-access-gtbz7\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4\" (UID: \"bf8c316f-51ce-4c69-865f-1d481ff30630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" Nov 03 16:14:05.669015 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.669001 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vdm\" (UniqueName: \"kubernetes.io/projected/fa01a744-9121-4f6a-b280-dba215c1ab8f-kube-api-access-d4vdm\") pod \"volume-data-source-validator-58c9f5f46d-88dnt\" (UID: \"fa01a744-9121-4f6a-b280-dba215c1ab8f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-88dnt" Nov 03 16:14:05.671986 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.671961 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phmbx\" (UniqueName: \"kubernetes.io/projected/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-kube-api-access-phmbx\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:05.677414 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.677389 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-bound-sa-token\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:05.680410 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.680390 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rznq\" (UniqueName: \"kubernetes.io/projected/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-kube-api-access-4rznq\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.714653 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.714626 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-j8t6k" Nov 03 16:14:05.727436 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.727395 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4j5k5" Nov 03 16:14:05.744881 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.744855 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-hxr8b" Nov 03 16:14:05.783510 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.783064 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:05.791214 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.790830 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-88dnt" Nov 03 16:14:05.803601 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.802779 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:05.808565 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.808540 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" Nov 03 16:14:05.817387 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.817195 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" Nov 03 16:14:05.828333 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.828108 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm2k2" Nov 03 16:14:05.834304 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:05.834280 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jdthd" Nov 03 16:14:06.033195 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.033166 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4j5k5"] Nov 03 16:14:06.034003 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:14:06.033968 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbc83f2_64b1_4804_a180_43df6459ffa4.slice/crio-9e34181ab541ebf6938d0e45793de04f3c6539adb7b03bedc6ca812ab2699f97 WatchSource:0}: Error finding container 9e34181ab541ebf6938d0e45793de04f3c6539adb7b03bedc6ca812ab2699f97: Status 404 returned error can't find the container with id 9e34181ab541ebf6938d0e45793de04f3c6539adb7b03bedc6ca812ab2699f97 Nov 03 16:14:06.036081 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:14:06.036053 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e9f545_8400_48b6_968d_eb65dc94b471.slice/crio-61b2a96cbbca5b98e23a41a60272db802884d84506232d4fa833e853770ebe30 WatchSource:0}: Error finding container 61b2a96cbbca5b98e23a41a60272db802884d84506232d4fa833e853770ebe30: Status 404 returned error can't find the container with id 61b2a96cbbca5b98e23a41a60272db802884d84506232d4fa833e853770ebe30 Nov 03 16:14:06.036260 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.036203 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-j8t6k"] Nov 03 16:14:06.066883 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:14:06.066841 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b370319_6de8_41e2_ae5f_21267a2fba74.slice/crio-4b29f2877c3d7d4b0259887e73ffe8e7d4895d8ebae92ff51273eb85de140587 WatchSource:0}: Error finding container 4b29f2877c3d7d4b0259887e73ffe8e7d4895d8ebae92ff51273eb85de140587: Status 404 returned error can't find the container with id 4b29f2877c3d7d4b0259887e73ffe8e7d4895d8ebae92ff51273eb85de140587 Nov 03 16:14:06.067787 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.067761 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-hxr8b"] Nov 03 16:14:06.139439 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.139402 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:06.139439 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.139450 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:06.139659 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.139491 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:06.139659 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.139539 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:06.139659 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.139575 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:06.139659 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:06.139599 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle podName:66b43213-9ec2-4164-9f8d-d7621bd57eb3 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:07.139580362 +0000 UTC m=+31.065485213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle") pod "router-default-5dc56589f5-wsj6w" (UID: "66b43213-9ec2-4164-9f8d-d7621bd57eb3") : configmap references non-existent config key: service-ca.crt Nov 03 16:14:06.139659 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:06.139631 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 16:14:06.139659 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.139656 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r755z\" (UID: \"2c812633-b15d-4a30-982d-80d276ec78c7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:06.139991 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.139678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:06.139991 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:06.139712 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:06.139991 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:06.139692 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls podName:5ced7a93-d9f0-44de-9a76-34a84e41843b nodeName:}" failed. No retries permitted until 2026-11-03 16:14:07.139674993 +0000 UTC m=+31.065579875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls") pod "dns-default-xfk72" (UID: "5ced7a93-d9f0-44de-9a76-34a84e41843b") : secret "dns-default-metrics-tls" not found Nov 03 16:14:06.139991 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:06.139727 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68c8fc97c8-vxzz9: secret "image-registry-tls" not found Nov 03 16:14:06.139991 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:06.139757 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 16:14:06.139991 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:06.139772 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls podName:a13e4eb2-b4f2-4f66-85d3-8bedb078823c nodeName:}" failed. No retries permitted until 2026-11-03 16:14:07.139760665 +0000 UTC m=+31.065665531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls") pod "image-registry-68c8fc97c8-vxzz9" (UID: "a13e4eb2-b4f2-4f66-85d3-8bedb078823c") : secret "image-registry-tls" not found Nov 03 16:14:06.139991 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:06.139807 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs podName:66b43213-9ec2-4164-9f8d-d7621bd57eb3 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:07.139781842 +0000 UTC m=+31.065686713 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs") pod "router-default-5dc56589f5-wsj6w" (UID: "66b43213-9ec2-4164-9f8d-d7621bd57eb3") : secret "router-metrics-certs-default" not found Nov 03 16:14:06.139991 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:06.139814 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 16:14:06.139991 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:06.139851 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls podName:c2ded168-1814-4922-9eee-00caf304b8af nodeName:}" failed. No retries permitted until 2026-11-03 16:14:07.139838918 +0000 UTC m=+31.065743790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vmcs4" (UID: "c2ded168-1814-4922-9eee-00caf304b8af") : secret "samples-operator-tls" not found Nov 03 16:14:06.139991 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:06.139852 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:06.139991 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:06.139867 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d945bd975-wtl6x: secret "image-registry-tls" not found Nov 03 16:14:06.139991 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:06.139903 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 16:14:06.139991 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.139847 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:06.139991 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:06.139904 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls podName:03cc593d-32bc-4453-8a45-2b546f862acf nodeName:}" failed. No retries permitted until 2026-11-03 16:14:07.139893343 +0000 UTC m=+31.065798217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls") pod "image-registry-7d945bd975-wtl6x" (UID: "03cc593d-32bc-4453-8a45-2b546f862acf") : secret "image-registry-tls" not found Nov 03 16:14:06.139991 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:06.139904 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:06.139991 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:06.139955 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert podName:2c812633-b15d-4a30-982d-80d276ec78c7 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:07.139944653 +0000 UTC m=+31.065849510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r755z" (UID: "2c812633-b15d-4a30-982d-80d276ec78c7") : secret "networking-console-plugin-cert" not found Nov 03 16:14:06.140455 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:06.139974 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls podName:e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:07.139965662 +0000 UTC m=+31.065870521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bw6bt" (UID: "e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28") : secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:06.150118 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.150042 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-88dnt"] Nov 03 16:14:06.151405 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:14:06.151376 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa01a744_9121_4f6a_b280_dba215c1ab8f.slice/crio-021004413f8c8b747965882c61d802c3df951d7f8ad43dc3f87794731f7d84c9 WatchSource:0}: Error finding container 021004413f8c8b747965882c61d802c3df951d7f8ad43dc3f87794731f7d84c9: Status 404 returned error can't find the container with id 021004413f8c8b747965882c61d802c3df951d7f8ad43dc3f87794731f7d84c9 Nov 03 16:14:06.173762 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.173536 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft"] Nov 03 16:14:06.174090 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:14:06.174053 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c106a2_ce74_4c57_80c1_b1f75da0d43d.slice/crio-b20754d576e2fb6d6e22e148d116e2c252bd0e552948343d86cdd6ff56741aca WatchSource:0}: Error finding container b20754d576e2fb6d6e22e148d116e2c252bd0e552948343d86cdd6ff56741aca: Status 404 returned error can't find the container with id b20754d576e2fb6d6e22e148d116e2c252bd0e552948343d86cdd6ff56741aca Nov 03 16:14:06.174959 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:14:06.174791 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1580792e_eb86_4d79_92a7_8055e6a8aaff.slice/crio-324ac350996ad1f2ce0d43d06952e2852e07aabc164151e5266c319e5df349c4 WatchSource:0}: Error finding container 324ac350996ad1f2ce0d43d06952e2852e07aabc164151e5266c319e5df349c4: Status 404 returned error can't find the container with id 324ac350996ad1f2ce0d43d06952e2852e07aabc164151e5266c319e5df349c4 Nov 03 16:14:06.175819 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.175796 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-964q9"] Nov 03 16:14:06.214405 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.214376 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-cm2k2"] Nov 03 16:14:06.214520 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.214437 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4"] Nov 03 16:14:06.215920 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.215896 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-tnh8w"] Nov 03 16:14:06.219649 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:14:06.219517 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216c0d66_a3ec_4748_b761_3420536bfa3a.slice/crio-1baf87fe253f97ace50fbc9aa73885717a8e523863b6a5af5a0c209c84a6226a WatchSource:0}: Error finding container 1baf87fe253f97ace50fbc9aa73885717a8e523863b6a5af5a0c209c84a6226a: Status 404 returned error can't find the container with id 1baf87fe253f97ace50fbc9aa73885717a8e523863b6a5af5a0c209c84a6226a Nov 03 16:14:06.266404 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.266383 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jdthd"] Nov 03 16:14:06.267583 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:14:06.267555 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f85dfa8_9158_437d_b151_d8eb8fe65b7b.slice/crio-bc768d9eab330521ad346bc708b210ee09b31e0192bcaf47f0532553dfd2f619 WatchSource:0}: Error finding container bc768d9eab330521ad346bc708b210ee09b31e0192bcaf47f0532553dfd2f619: Status 404 returned error can't find the container with id bc768d9eab330521ad346bc708b210ee09b31e0192bcaf47f0532553dfd2f619 Nov 03 16:14:06.881345 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.881304 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-964q9" event={"ID":"b9c106a2-ce74-4c57-80c1-b1f75da0d43d","Type":"ContainerStarted","Data":"b20754d576e2fb6d6e22e148d116e2c252bd0e552948343d86cdd6ff56741aca"} Nov 03 16:14:06.894725 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.893837 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-j8t6k" event={"ID":"78e9f545-8400-48b6-968d-eb65dc94b471","Type":"ContainerStarted","Data":"ed47d35a355940bf88b44d76f4711e811fccb8309e121984e48720e156436e2a"} Nov 03 16:14:06.894725 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.893943 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-j8t6k" event={"ID":"78e9f545-8400-48b6-968d-eb65dc94b471","Type":"ContainerStarted","Data":"61b2a96cbbca5b98e23a41a60272db802884d84506232d4fa833e853770ebe30"} Nov 03 16:14:06.898141 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.898102 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm2k2" event={"ID":"ba6c03a7-1ba9-48c0-b67b-6fb26263425e","Type":"ContainerStarted","Data":"33063db3a0abdf281cc0d58a01ecee734a14c03e79a2cd030a1be0adc5f398d7"} Nov 03 16:14:06.900919 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.900885 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jdthd" event={"ID":"8f85dfa8-9158-437d-b151-d8eb8fe65b7b","Type":"ContainerStarted","Data":"bc768d9eab330521ad346bc708b210ee09b31e0192bcaf47f0532553dfd2f619"} Nov 03 16:14:06.904238 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.904126 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" event={"ID":"1580792e-eb86-4d79-92a7-8055e6a8aaff","Type":"ContainerStarted","Data":"324ac350996ad1f2ce0d43d06952e2852e07aabc164151e5266c319e5df349c4"} Nov 03 16:14:06.907459 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.907374 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" event={"ID":"bf8c316f-51ce-4c69-865f-1d481ff30630","Type":"ContainerStarted","Data":"4c7564eb1c2844cff816a627465bb36ce2d4260057a1b479a34de3a577540c15"} Nov 03 16:14:06.910278 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.910251 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" event={"ID":"216c0d66-a3ec-4748-b761-3420536bfa3a","Type":"ContainerStarted","Data":"1baf87fe253f97ace50fbc9aa73885717a8e523863b6a5af5a0c209c84a6226a"} Nov 03 16:14:06.919639 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.919599 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-88dnt" event={"ID":"fa01a744-9121-4f6a-b280-dba215c1ab8f","Type":"ContainerStarted","Data":"021004413f8c8b747965882c61d802c3df951d7f8ad43dc3f87794731f7d84c9"} Nov 03 16:14:06.925114 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.925086 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hxr8b" event={"ID":"4b370319-6de8-41e2-ae5f-21267a2fba74","Type":"ContainerStarted","Data":"058a2d3727b1f09e024781cbeefe1001ced1751abbebe90542e0d5d4e860f866"} Nov 03 16:14:06.925239 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.925136 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-hxr8b" event={"ID":"4b370319-6de8-41e2-ae5f-21267a2fba74","Type":"ContainerStarted","Data":"4b29f2877c3d7d4b0259887e73ffe8e7d4895d8ebae92ff51273eb85de140587"} Nov 03 16:14:06.932497 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.932465 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4j5k5" event={"ID":"4cbc83f2-64b1-4804-a180-43df6459ffa4","Type":"ContainerStarted","Data":"5de23bea225fa7ce35f2861d69d78b7ad0f5f20feeacebb8c9625f7c25195048"} Nov 03 16:14:06.932632 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.932521 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4j5k5" event={"ID":"4cbc83f2-64b1-4804-a180-43df6459ffa4","Type":"ContainerStarted","Data":"9e34181ab541ebf6938d0e45793de04f3c6539adb7b03bedc6ca812ab2699f97"} Nov 03 16:14:06.942879 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:06.942824 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-j8t6k" podStartSLOduration=210.942807436 podStartE2EDuration="3m30.942807436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:06.942398653 +0000 UTC m=+30.868303529" watchObservedRunningTime="2026-11-03 16:14:06.942807436 +0000 UTC m=+30.868712310" Nov 03 16:14:07.046823 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:07.046757 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-4j5k5" podStartSLOduration=211.04673602 podStartE2EDuration="3m31.04673602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:07.044788994 +0000 UTC m=+30.970693868" watchObservedRunningTime="2026-11-03 16:14:07.04673602 +0000 UTC m=+30.972640899" Nov 03 16:14:07.047014 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:07.046913 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-hxr8b" podStartSLOduration=211.046905878 podStartE2EDuration="3m31.046905878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:06.996936271 +0000 UTC m=+30.922841147" watchObservedRunningTime="2026-11-03 16:14:07.046905878 +0000 UTC m=+30.972810752" Nov 03 16:14:07.149428 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:07.149313 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r755z\" (UID: \"2c812633-b15d-4a30-982d-80d276ec78c7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:07.149428 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:07.149364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:07.149428 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:07.149424 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:07.149661 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:07.149497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:07.149661 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:07.149516 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:07.149661 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:07.149541 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:07.149661 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:07.149563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:07.149661 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:07.149583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:07.149832 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:07.149679 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:07.149832 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:07.149690 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68c8fc97c8-vxzz9: secret "image-registry-tls" not found Nov 03 16:14:07.149832 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:07.149784 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls podName:a13e4eb2-b4f2-4f66-85d3-8bedb078823c nodeName:}" failed. No retries permitted until 2026-11-03 16:14:09.149751674 +0000 UTC m=+33.075656527 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls") pod "image-registry-68c8fc97c8-vxzz9" (UID: "a13e4eb2-b4f2-4f66-85d3-8bedb078823c") : secret "image-registry-tls" not found Nov 03 16:14:07.150236 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:07.150218 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 16:14:07.150323 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:07.150265 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert podName:2c812633-b15d-4a30-982d-80d276ec78c7 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:09.150253869 +0000 UTC m=+33.076158720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r755z" (UID: "2c812633-b15d-4a30-982d-80d276ec78c7") : secret "networking-console-plugin-cert" not found Nov 03 16:14:07.150323 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:07.150304 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 16:14:07.150418 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:07.150325 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs podName:66b43213-9ec2-4164-9f8d-d7621bd57eb3 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:09.150318683 +0000 UTC m=+33.076223534 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs") pod "router-default-5dc56589f5-wsj6w" (UID: "66b43213-9ec2-4164-9f8d-d7621bd57eb3") : secret "router-metrics-certs-default" not found Nov 03 16:14:07.150418 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:07.150364 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:07.150418 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:07.150393 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls podName:e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:09.150385028 +0000 UTC m=+33.076289879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bw6bt" (UID: "e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28") : secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:07.150558 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:07.150436 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle podName:66b43213-9ec2-4164-9f8d-d7621bd57eb3 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:09.15042929 +0000 UTC m=+33.076334141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle") pod "router-default-5dc56589f5-wsj6w" (UID: "66b43213-9ec2-4164-9f8d-d7621bd57eb3") : configmap references non-existent config key: service-ca.crt Nov 03 16:14:07.150558 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:07.150472 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 16:14:07.150558 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:07.150489 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls podName:5ced7a93-d9f0-44de-9a76-34a84e41843b nodeName:}" failed. No retries permitted until 2026-11-03 16:14:09.150484189 +0000 UTC m=+33.076389040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls") pod "dns-default-xfk72" (UID: "5ced7a93-d9f0-44de-9a76-34a84e41843b") : secret "dns-default-metrics-tls" not found Nov 03 16:14:07.150558 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:07.150522 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 16:14:07.150558 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:07.150548 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls podName:c2ded168-1814-4922-9eee-00caf304b8af nodeName:}" failed. No retries permitted until 2026-11-03 16:14:09.150541999 +0000 UTC m=+33.076446849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vmcs4" (UID: "c2ded168-1814-4922-9eee-00caf304b8af") : secret "samples-operator-tls" not found Nov 03 16:14:07.150826 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:07.150584 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:07.150826 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:07.150591 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d945bd975-wtl6x: secret "image-registry-tls" not found Nov 03 16:14:07.150826 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:07.150616 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls podName:03cc593d-32bc-4453-8a45-2b546f862acf nodeName:}" failed. No retries permitted until 2026-11-03 16:14:09.150609302 +0000 UTC m=+33.076514156 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls") pod "image-registry-7d945bd975-wtl6x" (UID: "03cc593d-32bc-4453-8a45-2b546f862acf") : secret "image-registry-tls" not found Nov 03 16:14:08.955392 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:08.955353 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-88dnt" event={"ID":"fa01a744-9121-4f6a-b280-dba215c1ab8f","Type":"ContainerStarted","Data":"0c6e686e6f4497fe681c3ecddb44b4e151df789166727d7b6ad3350bd98cf568"} Nov 03 16:14:09.003733 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:09.003655 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-88dnt" podStartSLOduration=181.761574326 podStartE2EDuration="3m4.003639169s" totalImagesPullingTime="2.242064843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:09.001596119 +0000 UTC m=+32.927500992" watchObservedRunningTime="2026-11-03 16:14:09.003639169 +0000 UTC m=+32.929544042" Nov 03 16:14:09.170455 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:09.170379 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:09.171470 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:09.170713 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:09.171470 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:09.170773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:09.171470 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:09.170817 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:09.171470 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:09.170849 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:09.171470 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:09.170895 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r755z\" (UID: \"2c812633-b15d-4a30-982d-80d276ec78c7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:09.171470 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:09.170924 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:09.171470 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:09.170984 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:09.171470 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:09.171122 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:09.171470 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:09.171179 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls podName:e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:13.171160228 +0000 UTC m=+37.097065093 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bw6bt" (UID: "e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28") : secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:09.171470 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:09.171247 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 16:14:09.171470 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:09.171276 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls podName:5ced7a93-d9f0-44de-9a76-34a84e41843b nodeName:}" failed. No retries permitted until 2026-11-03 16:14:13.171266465 +0000 UTC m=+37.097171329 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls") pod "dns-default-xfk72" (UID: "5ced7a93-d9f0-44de-9a76-34a84e41843b") : secret "dns-default-metrics-tls" not found Nov 03 16:14:09.171470 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:09.171317 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:09.171470 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:09.171333 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d945bd975-wtl6x: secret "image-registry-tls" not found Nov 03 16:14:09.171470 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:09.171341 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:09.171470 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:09.171352 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68c8fc97c8-vxzz9: secret "image-registry-tls" not found Nov 03 16:14:09.171470 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:09.171372 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls podName:03cc593d-32bc-4453-8a45-2b546f862acf nodeName:}" failed. No retries permitted until 2026-11-03 16:14:13.171357948 +0000 UTC m=+37.097262799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls") pod "image-registry-7d945bd975-wtl6x" (UID: "03cc593d-32bc-4453-8a45-2b546f862acf") : secret "image-registry-tls" not found Nov 03 16:14:09.172294 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:09.171389 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls podName:a13e4eb2-b4f2-4f66-85d3-8bedb078823c nodeName:}" failed. No retries permitted until 2026-11-03 16:14:13.171380192 +0000 UTC m=+37.097285042 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls") pod "image-registry-68c8fc97c8-vxzz9" (UID: "a13e4eb2-b4f2-4f66-85d3-8bedb078823c") : secret "image-registry-tls" not found Nov 03 16:14:09.172294 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:09.171404 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle podName:66b43213-9ec2-4164-9f8d-d7621bd57eb3 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:13.171395834 +0000 UTC m=+37.097300761 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle") pod "router-default-5dc56589f5-wsj6w" (UID: "66b43213-9ec2-4164-9f8d-d7621bd57eb3") : configmap references non-existent config key: service-ca.crt Nov 03 16:14:09.172294 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:09.171423 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 16:14:09.172294 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:09.171464 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 16:14:09.172294 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:09.171499 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls podName:c2ded168-1814-4922-9eee-00caf304b8af nodeName:}" failed. No retries permitted until 2026-11-03 16:14:13.171489843 +0000 UTC m=+37.097394696 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vmcs4" (UID: "c2ded168-1814-4922-9eee-00caf304b8af") : secret "samples-operator-tls" not found Nov 03 16:14:09.172294 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:09.171527 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 16:14:09.172294 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:09.171665 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs podName:66b43213-9ec2-4164-9f8d-d7621bd57eb3 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:13.171553626 +0000 UTC m=+37.097458491 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs") pod "router-default-5dc56589f5-wsj6w" (UID: "66b43213-9ec2-4164-9f8d-d7621bd57eb3") : secret "router-metrics-certs-default" not found Nov 03 16:14:09.172833 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:09.172812 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert podName:2c812633-b15d-4a30-982d-80d276ec78c7 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:13.172793109 +0000 UTC m=+37.098697974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r755z" (UID: "2c812633-b15d-4a30-982d-80d276ec78c7") : secret "networking-console-plugin-cert" not found Nov 03 16:14:10.386367 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:10.386322 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:14:10.386811 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:10.386372 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:14:10.387170 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:10.387142 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 16:14:10.387278 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:10.387220 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs podName:55767581-2059-49bf-8dd0-69e31d5adb83 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:42.387198981 +0000 UTC m=+66.313103835 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs") pod "network-metrics-daemon-n7hw7" (UID: "55767581-2059-49bf-8dd0-69e31d5adb83") : secret "metrics-daemon-secret" not found Nov 03 16:14:10.391200 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:10.391175 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:14:10.403312 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:10.403275 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:14:10.487189 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:10.487149 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9slx\" (UniqueName: \"kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx\") pod \"network-check-target-5fq8q\" (UID: \"110bff76-1581-4379-8468-441a3a81dea2\") " pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:14:10.490502 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:10.490469 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9slx\" (UniqueName: \"kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx\") pod \"network-check-target-5fq8q\" (UID: \"110bff76-1581-4379-8468-441a3a81dea2\") " pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:14:10.693738 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:10.693502 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:14:10.866075 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:10.866045 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p7vvq"] Nov 03 16:14:10.921815 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:14:10.921771 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c96f345_2cda_4554_bef3_be416dc50225.slice/crio-3938c3ec0d18e946ab09733a2e7c128bec132fe52206ebc22737c77e423812a1 WatchSource:0}: Error finding container 3938c3ec0d18e946ab09733a2e7c128bec132fe52206ebc22737c77e423812a1: Status 404 returned error can't find the container with id 3938c3ec0d18e946ab09733a2e7c128bec132fe52206ebc22737c77e423812a1 Nov 03 16:14:10.966345 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:10.966279 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/0.log" Nov 03 16:14:10.966345 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:10.966317 2577 generic.go:356] "Generic (PLEG): container finished" podID="216c0d66-a3ec-4748-b761-3420536bfa3a" containerID="b6a4c0afefec9e27954d3638d1de129e864ae3a5965fc6253df65cc0f36fc629" exitCode=255 Nov 03 16:14:10.966543 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:10.966350 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" event={"ID":"216c0d66-a3ec-4748-b761-3420536bfa3a","Type":"ContainerDied","Data":"b6a4c0afefec9e27954d3638d1de129e864ae3a5965fc6253df65cc0f36fc629"} Nov 03 16:14:10.966651 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:10.966632 2577 scope.go:122] "RemoveContainer" containerID="b6a4c0afefec9e27954d3638d1de129e864ae3a5965fc6253df65cc0f36fc629" Nov 03 16:14:10.969240 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:10.969187 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-964q9" event={"ID":"b9c106a2-ce74-4c57-80c1-b1f75da0d43d","Type":"ContainerStarted","Data":"087377bd323389ac129c34915b45f033a9599717a55cf079aa2b973df0b1983c"} Nov 03 16:14:10.970454 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:10.970431 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p7vvq" event={"ID":"8c96f345-2cda-4554-bef3-be416dc50225","Type":"ContainerStarted","Data":"3938c3ec0d18e946ab09733a2e7c128bec132fe52206ebc22737c77e423812a1"} Nov 03 16:14:11.062538 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.062480 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-964q9" podStartSLOduration=181.66893502 podStartE2EDuration="3m6.062462753s" totalImagesPullingTime="4.393527733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:11.061680947 +0000 UTC m=+34.987585820" watchObservedRunningTime="2026-11-03 16:14:11.062462753 +0000 UTC m=+34.988367626" Nov 03 16:14:11.436458 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.436434 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rj5rd"] Nov 03 16:14:11.467732 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.467216 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:11.468245 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.468223 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rj5rd"] Nov 03 16:14:11.571722 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.571472 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 16:14:11.571722 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.571557 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 16:14:11.571722 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.571668 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5snks\"" Nov 03 16:14:11.599537 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.599501 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f02c434e-ede0-4bea-8830-f3444c21a6a6-data-volume\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:11.599676 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.599552 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:11.599676 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.599588 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfkpb\" (UniqueName: \"kubernetes.io/projected/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-api-access-kfkpb\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:11.599676 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.599621 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:11.599867 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.599747 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f02c434e-ede0-4bea-8830-f3444c21a6a6-crio-socket\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:11.699028 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.698996 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5fq8q"] Nov 03 16:14:11.700355 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.700324 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f02c434e-ede0-4bea-8830-f3444c21a6a6-crio-socket\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:11.700510 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.700469 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f02c434e-ede0-4bea-8830-f3444c21a6a6-data-volume\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:11.700600 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.700581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:11.700796 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.700643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfkpb\" (UniqueName: \"kubernetes.io/projected/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-api-access-kfkpb\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:11.700796 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.700691 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:11.700906 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:11.700874 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 16:14:11.700962 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:11.700941 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls podName:f02c434e-ede0-4bea-8830-f3444c21a6a6 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:12.200923609 +0000 UTC m=+36.126828476 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rj5rd" (UID: "f02c434e-ede0-4bea-8830-f3444c21a6a6") : secret "insights-runtime-extractor-tls" not found Nov 03 16:14:11.701235 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.701214 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f02c434e-ede0-4bea-8830-f3444c21a6a6-crio-socket\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:11.701575 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.701557 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f02c434e-ede0-4bea-8830-f3444c21a6a6-data-volume\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:11.701644 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:11.701630 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 16:14:11.701716 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:11.701679 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-rbac-proxy-cm podName:f02c434e-ede0-4bea-8830-f3444c21a6a6 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:12.201660006 +0000 UTC m=+36.127564872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rj5rd" (UID: "f02c434e-ede0-4bea-8830-f3444c21a6a6") : configmap "kube-rbac-proxy" not found Nov 03 16:14:11.730427 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.730393 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfkpb\" (UniqueName: \"kubernetes.io/projected/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-api-access-kfkpb\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:11.747449 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:14:11.747408 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110bff76_1581_4379_8468_441a3a81dea2.slice/crio-385883e4ba88316132665e384730bd687c493c1006cfdbcc39d7ac1b870ed4bf WatchSource:0}: Error finding container 385883e4ba88316132665e384730bd687c493c1006cfdbcc39d7ac1b870ed4bf: Status 404 returned error can't find the container with id 385883e4ba88316132665e384730bd687c493c1006cfdbcc39d7ac1b870ed4bf Nov 03 16:14:11.976762 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.976647 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5fq8q" event={"ID":"110bff76-1581-4379-8468-441a3a81dea2","Type":"ContainerStarted","Data":"385883e4ba88316132665e384730bd687c493c1006cfdbcc39d7ac1b870ed4bf"} Nov 03 16:14:11.978381 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.978353 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm2k2" event={"ID":"ba6c03a7-1ba9-48c0-b67b-6fb26263425e","Type":"ContainerStarted","Data":"0077017a6fd8979d0aea2a3c93700bbe514d552dd49ec868ff5c5c25a22de229"} Nov 03 16:14:11.980096 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.980044 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jdthd" event={"ID":"8f85dfa8-9158-437d-b151-d8eb8fe65b7b","Type":"ContainerStarted","Data":"359730592b4ae758c4a3cfc96e7f756c132920235fa7a3aa5ac02b2286552bf7"} Nov 03 16:14:11.981756 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.981732 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" event={"ID":"1580792e-eb86-4d79-92a7-8055e6a8aaff","Type":"ContainerStarted","Data":"30a9ebb9c659c99b128358965f83114a59d103a42daf52c9a50fa1427b14e76f"} Nov 03 16:14:11.984018 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.983594 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" event={"ID":"bf8c316f-51ce-4c69-865f-1d481ff30630","Type":"ContainerStarted","Data":"622c06122a7b144c63fe80289c612a76c433e21a9d36aeab2e9ac514329c1fb4"} Nov 03 16:14:11.986084 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.986061 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/1.log" Nov 03 16:14:11.986487 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.986467 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/0.log" Nov 03 16:14:11.986588 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.986508 2577 generic.go:356] "Generic (PLEG): container finished" podID="216c0d66-a3ec-4748-b761-3420536bfa3a" containerID="02633c14002a1b44b88bd7bf4802aa104ef3c6c904de369370b2ccfa72bbf740" exitCode=255 Nov 03 16:14:11.986588 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.986576 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" event={"ID":"216c0d66-a3ec-4748-b761-3420536bfa3a","Type":"ContainerDied","Data":"02633c14002a1b44b88bd7bf4802aa104ef3c6c904de369370b2ccfa72bbf740"} Nov 03 16:14:11.986724 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.986618 2577 scope.go:122] "RemoveContainer" containerID="b6a4c0afefec9e27954d3638d1de129e864ae3a5965fc6253df65cc0f36fc629" Nov 03 16:14:11.986845 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:11.986813 2577 scope.go:122] "RemoveContainer" containerID="02633c14002a1b44b88bd7bf4802aa104ef3c6c904de369370b2ccfa72bbf740" Nov 03 16:14:11.987194 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:11.987019 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tnh8w_openshift-console-operator(216c0d66-a3ec-4748-b761-3420536bfa3a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" podUID="216c0d66-a3ec-4748-b761-3420536bfa3a" Nov 03 16:14:12.025219 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.025157 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm2k2" podStartSLOduration=177.458980179 podStartE2EDuration="3m3.025136032s" totalImagesPullingTime="5.566155853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:12.024420036 +0000 UTC m=+35.950324910" watchObservedRunningTime="2026-11-03 16:14:12.025136032 +0000 UTC m=+35.951040910" Nov 03 16:14:12.082529 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.082464 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jdthd" podStartSLOduration=1.936955431 podStartE2EDuration="7.082443321s" totalImagesPullingTime="5.14548789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:12.080762368 +0000 UTC m=+36.006667242" watchObservedRunningTime="2026-11-03 16:14:12.082443321 +0000 UTC m=+36.008348198" Nov 03 16:14:12.129721 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.129642 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" podStartSLOduration=181.957654898 podStartE2EDuration="3m7.129625633s" totalImagesPullingTime="5.171970735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:12.128095474 +0000 UTC m=+36.054000348" watchObservedRunningTime="2026-11-03 16:14:12.129625633 +0000 UTC m=+36.055530507" Nov 03 16:14:12.183510 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.183266 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" podStartSLOduration=181.97072853 podStartE2EDuration="3m7.183243484s" totalImagesPullingTime="5.212514954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:12.176932217 +0000 UTC m=+36.102837090" watchObservedRunningTime="2026-11-03 16:14:12.183243484 +0000 UTC m=+36.109148357" Nov 03 16:14:12.210026 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.209979 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:12.210176 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.210044 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:12.210250 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:12.210226 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 16:14:12.210316 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:12.210298 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls podName:f02c434e-ede0-4bea-8830-f3444c21a6a6 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:13.210279812 +0000 UTC m=+37.136184681 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rj5rd" (UID: "f02c434e-ede0-4bea-8830-f3444c21a6a6") : secret "insights-runtime-extractor-tls" not found Nov 03 16:14:12.210735 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:12.210688 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 16:14:12.210825 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:12.210774 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-rbac-proxy-cm podName:f02c434e-ede0-4bea-8830-f3444c21a6a6 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:13.210757235 +0000 UTC m=+37.136662101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rj5rd" (UID: "f02c434e-ede0-4bea-8830-f3444c21a6a6") : configmap "kube-rbac-proxy" not found Nov 03 16:14:12.518472 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.518434 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-g8vzv"] Nov 03 16:14:12.548334 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.548256 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-g8vzv"] Nov 03 16:14:12.548457 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.548404 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g8vzv" Nov 03 16:14:12.616799 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.616639 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 16:14:12.616799 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.616772 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 16:14:12.641877 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.641842 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6xtgh\"" Nov 03 16:14:12.715856 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.715819 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d2cr\" (UniqueName: \"kubernetes.io/projected/94fb68fc-92af-4e72-824b-7092f9875560-kube-api-access-2d2cr\") pod \"migrator-66df5d496b-g8vzv\" (UID: \"94fb68fc-92af-4e72-824b-7092f9875560\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g8vzv" Nov 03 16:14:12.817059 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.816968 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d2cr\" (UniqueName: \"kubernetes.io/projected/94fb68fc-92af-4e72-824b-7092f9875560-kube-api-access-2d2cr\") pod \"migrator-66df5d496b-g8vzv\" (UID: \"94fb68fc-92af-4e72-824b-7092f9875560\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g8vzv" Nov 03 16:14:12.843284 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.843249 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d2cr\" (UniqueName: \"kubernetes.io/projected/94fb68fc-92af-4e72-824b-7092f9875560-kube-api-access-2d2cr\") pod \"migrator-66df5d496b-g8vzv\" (UID: \"94fb68fc-92af-4e72-824b-7092f9875560\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g8vzv" Nov 03 16:14:12.880264 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.880229 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g8vzv" Nov 03 16:14:12.993732 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.993691 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/1.log" Nov 03 16:14:12.994100 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.994080 2577 scope.go:122] "RemoveContainer" containerID="02633c14002a1b44b88bd7bf4802aa104ef3c6c904de369370b2ccfa72bbf740" Nov 03 16:14:12.994323 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:12.994290 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tnh8w_openshift-console-operator(216c0d66-a3ec-4748-b761-3420536bfa3a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" podUID="216c0d66-a3ec-4748-b761-3420536bfa3a" Nov 03 16:14:12.996307 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:12.995937 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5fq8q" event={"ID":"110bff76-1581-4379-8468-441a3a81dea2","Type":"ContainerStarted","Data":"624b655fb6ef35d39ce13f10fd1c9e258c206db29990b5ed50427c426f90a586"} Nov 03 16:14:13.091156 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:13.091059 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5fq8q" podStartSLOduration=35.576369919 podStartE2EDuration="36.091042835s" totalImagesPullingTime="514.672916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:13:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:13.089621696 +0000 UTC m=+37.015526570" watchObservedRunningTime="2026-11-03 16:14:13.091042835 +0000 UTC m=+37.016947712" Nov 03 16:14:13.212615 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:13.212590 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-g8vzv"] Nov 03 16:14:13.214244 ip-10-0-143-64 kubenswrapper[2577]: W1103 16:14:13.214204 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fb68fc_92af_4e72_824b_7092f9875560.slice/crio-73372b2c138ebb0b5a2ce8af6592946e65bd17ddcd56ecbb1b40093dc8c02fea WatchSource:0}: Error finding container 73372b2c138ebb0b5a2ce8af6592946e65bd17ddcd56ecbb1b40093dc8c02fea: Status 404 returned error can't find the container with id 73372b2c138ebb0b5a2ce8af6592946e65bd17ddcd56ecbb1b40093dc8c02fea Nov 03 16:14:13.222652 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:13.222623 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:13.222758 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:13.222679 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:13.222758 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:13.222739 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:13.222879 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:13.222778 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r755z\" (UID: \"2c812633-b15d-4a30-982d-80d276ec78c7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:13.222879 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.222783 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 16:14:13.222879 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:13.222806 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:13.222879 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:13.222831 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:13.222879 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.222848 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls podName:c2ded168-1814-4922-9eee-00caf304b8af nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.222827899 +0000 UTC m=+45.148732762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vmcs4" (UID: "c2ded168-1814-4922-9eee-00caf304b8af") : secret "samples-operator-tls" not found Nov 03 16:14:13.223046 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.222898 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 16:14:13.223046 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:13.222898 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:13.223046 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.222943 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs podName:66b43213-9ec2-4164-9f8d-d7621bd57eb3 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.222931341 +0000 UTC m=+45.148836205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs") pod "router-default-5dc56589f5-wsj6w" (UID: "66b43213-9ec2-4164-9f8d-d7621bd57eb3") : secret "router-metrics-certs-default" not found Nov 03 16:14:13.223046 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:13.222969 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:13.223046 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.222983 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 16:14:13.223046 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.223017 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls podName:f02c434e-ede0-4bea-8830-f3444c21a6a6 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:15.223006246 +0000 UTC m=+39.148911114 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rj5rd" (UID: "f02c434e-ede0-4bea-8830-f3444c21a6a6") : secret "insights-runtime-extractor-tls" not found Nov 03 16:14:13.223046 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.223042 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:13.223276 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:13.223063 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:13.223276 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.223076 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls podName:e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.223065845 +0000 UTC m=+45.148970722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bw6bt" (UID: "e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28") : secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:13.223276 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:13.223100 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:13.223276 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.223135 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:13.223276 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.223146 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d945bd975-wtl6x: secret "image-registry-tls" not found Nov 03 16:14:13.223276 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.223177 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls podName:03cc593d-32bc-4453-8a45-2b546f862acf nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.22316852 +0000 UTC m=+45.149073392 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls") pod "image-registry-7d945bd975-wtl6x" (UID: "03cc593d-32bc-4453-8a45-2b546f862acf") : secret "image-registry-tls" not found Nov 03 16:14:13.223276 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.223193 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 16:14:13.223276 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.223223 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls podName:5ced7a93-d9f0-44de-9a76-34a84e41843b nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.223212888 +0000 UTC m=+45.149117744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls") pod "dns-default-xfk72" (UID: "5ced7a93-d9f0-44de-9a76-34a84e41843b") : secret "dns-default-metrics-tls" not found Nov 03 16:14:13.223276 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.223229 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:13.223276 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.223238 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68c8fc97c8-vxzz9: secret "image-registry-tls" not found Nov 03 16:14:13.223276 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.223267 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls podName:a13e4eb2-b4f2-4f66-85d3-8bedb078823c nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.2232567 +0000 UTC m=+45.149161574 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls") pod "image-registry-68c8fc97c8-vxzz9" (UID: "a13e4eb2-b4f2-4f66-85d3-8bedb078823c") : secret "image-registry-tls" not found Nov 03 16:14:13.223644 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.223313 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle podName:66b43213-9ec2-4164-9f8d-d7621bd57eb3 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.223301953 +0000 UTC m=+45.149206819 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle") pod "router-default-5dc56589f5-wsj6w" (UID: "66b43213-9ec2-4164-9f8d-d7621bd57eb3") : configmap references non-existent config key: service-ca.crt Nov 03 16:14:13.223644 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.223318 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 16:14:13.223644 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.223338 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert podName:2c812633-b15d-4a30-982d-80d276ec78c7 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.223332566 +0000 UTC m=+45.149237468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r755z" (UID: "2c812633-b15d-4a30-982d-80d276ec78c7") : secret "networking-console-plugin-cert" not found Nov 03 16:14:13.223644 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.223354 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 16:14:13.223644 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:13.223386 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-rbac-proxy-cm podName:f02c434e-ede0-4bea-8830-f3444c21a6a6 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:15.223376182 +0000 UTC m=+39.149281136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rj5rd" (UID: "f02c434e-ede0-4bea-8830-f3444c21a6a6") : configmap "kube-rbac-proxy" not found Nov 03 16:14:13.999978 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:13.999938 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g8vzv" event={"ID":"94fb68fc-92af-4e72-824b-7092f9875560","Type":"ContainerStarted","Data":"73372b2c138ebb0b5a2ce8af6592946e65bd17ddcd56ecbb1b40093dc8c02fea"} Nov 03 16:14:14.000373 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:14.000060 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:14:15.011301 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:15.011247 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g8vzv" event={"ID":"94fb68fc-92af-4e72-824b-7092f9875560","Type":"ContainerStarted","Data":"318e3fd5f3bb1f3515e4a4b341ba7a42a84d30321c811449da9d515d38efaf10"} Nov 03 16:14:15.058632 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:15.058575 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g8vzv" podStartSLOduration=1.603212326 podStartE2EDuration="3.058558401s" totalImagesPullingTime="1.455346075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:15.05733269 +0000 UTC m=+38.983237565" watchObservedRunningTime="2026-11-03 16:14:15.058558401 +0000 UTC m=+38.984463275" Nov 03 16:14:15.245141 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:15.245094 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:15.245344 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:15.245265 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 16:14:15.245344 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:15.245331 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:15.245459 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:15.245352 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls podName:f02c434e-ede0-4bea-8830-f3444c21a6a6 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:19.245329514 +0000 UTC m=+43.171234378 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rj5rd" (UID: "f02c434e-ede0-4bea-8830-f3444c21a6a6") : secret "insights-runtime-extractor-tls" not found Nov 03 16:14:15.245459 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:15.245405 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 16:14:15.245583 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:15.245467 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-rbac-proxy-cm podName:f02c434e-ede0-4bea-8830-f3444c21a6a6 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:19.245451902 +0000 UTC m=+43.171356754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rj5rd" (UID: "f02c434e-ede0-4bea-8830-f3444c21a6a6") : configmap "kube-rbac-proxy" not found Nov 03 16:14:15.803314 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:15.803280 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:15.803472 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:15.803333 2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:15.803648 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:15.803637 2577 scope.go:122] "RemoveContainer" containerID="02633c14002a1b44b88bd7bf4802aa104ef3c6c904de369370b2ccfa72bbf740" Nov 03 16:14:15.803838 ip-10-0-143-64 kubenswrapper[2577]: E1103 16:14:15.803820 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tnh8w_openshift-console-operator(216c0d66-a3ec-4748-b761-3420536bfa3a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" podUID="216c0d66-a3ec-4748-b761-3420536bfa3a" Nov 03 16:14:16.317180 ip-10-0-143-64 kubenswrapper[2577]: I1103 16:14:16.317030 2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:14:16.317150 ip-10-0-143-64 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 16:14:16.334150 ip-10-0-143-64 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 16:14:16.334332 ip-10-0-143-64 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 16:14:16.334520 ip-10-0-143-64 systemd[1]: kubelet.service: Consumed 3.224s CPU time. Nov 03 16:14:16.357332 ip-10-0-143-64 systemd[1]: Starting Kubernetes Kubelet... Nov 03 16:14:16.477131 ip-10-0-143-64 kubenswrapper[4947]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:14:16.477131 ip-10-0-143-64 kubenswrapper[4947]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 16:14:16.477131 ip-10-0-143-64 kubenswrapper[4947]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:14:16.477131 ip-10-0-143-64 kubenswrapper[4947]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 16:14:16.479260 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479169 4947 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:14:16.479260 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479259 4947 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479264 4947 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479267 4947 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479270 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479272 4947 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479275 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479278 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479280 4947 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479283 4947 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479285 4947 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479288 4947 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479290 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479293 4947 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479296 4947 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479298 4947 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479301 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479303 4947 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479306 4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479308 4947 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:14:16.479333 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479311 4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479314 4947 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479317 4947 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479319 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479322 4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479324 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479327 4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479335 4947 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479338 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479341 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479344 4947 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479347 4947 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479349 4947 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479352 4947 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479355 4947 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479357 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479360 4947 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479362 4947 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479365 4947 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:14:16.479840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479367 4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479370 4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479372 4947 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479375 4947 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479378 4947 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479380 4947 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479382 4947 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479386 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479389 4947 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479392 4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479395 4947 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479399 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479401 4947 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479404 4947 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479406 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479409 4947 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479412 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479415 4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479419 4947 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:14:16.480315 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479423 4947 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479425 4947 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479428 4947 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479431 4947 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479434 4947 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479437 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479440 4947 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479442 4947 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479445 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479448 4947 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479450 4947 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479453 4947 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479456 4947 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479458 4947 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479461 4947 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479464 4947 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479467 4947 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479469 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479472 4947 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479475 4947 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:14:16.480821 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479477 4947 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479480 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479483 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479485 4947 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479487 4947 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479490 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479494 4947 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479498 4947 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479501 4947 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479504 4947 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479507 4947 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479510 4947 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479512 4947 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479515 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479517 4947 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479520 4947 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479522 4947 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479525 4947 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479527 4947 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:14:16.481317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479530 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479532 4947 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479535 4947 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479537 4947 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479540 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479542 4947 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479545 4947 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479548 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.479550 4947 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480435 4947 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480444 4947 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480448 4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480451 4947 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480454 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480457 4947 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480460 4947 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480463 4947 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480465 4947 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480468 4947 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:14:16.481842 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480471 4947 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480473 4947 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480476 4947 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480478 4947 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480481 4947 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480483 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480486 4947 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480488 4947 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480491 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480493 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480496 4947 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480499 4947 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480502 4947 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480504 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480507 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480516 4947 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480519 4947 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480521 4947 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480524 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:14:16.482369 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480527 4947 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480529 4947 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480532 4947 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480540 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480543 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480545 4947 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480548 4947 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480550 4947 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480553 4947 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480556 4947 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480558 4947 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480561 4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480563 4947 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480566 4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480568 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480571 4947 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480573 4947 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480575 4947 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480578 4947 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480580 4947 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:14:16.482865 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480583 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480585 4947 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480587 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480596 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480599 4947 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480601 4947 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480604 4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480606 4947 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480609 4947 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480612 4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480614 4947 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480617 4947 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480619 4947 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480622 4947 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480624 4947 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480626 4947 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480635 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480637 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480640 4947 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480642 4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:14:16.483362 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480645 4947 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480648 4947 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480650 4947 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480653 4947 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480655 4947 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480669 4947 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480671 4947 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480674 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480676 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480679 4947 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480682 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480684 4947 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480687 4947 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480689 4947 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480692 4947 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480709 4947 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480712 4947 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480715 4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480717 4947 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:14:16.483892 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480721 4947 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480725 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480728 4947 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480731 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480734 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480737 4947 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480740 4947 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480742 4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480744 4947 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480747 4947 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480756 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480759 4947 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480761 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480764 4947 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480767 4947 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480770 4947 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480773 4947 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.480775 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481303 4947 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481308 4947 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:14:16.484420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481310 4947 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481313 4947 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481315 4947 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481318 4947 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481321 4947 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481324 4947 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481326 4947 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481329 4947 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481332 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481335 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481339 4947 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481343 4947 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481346 4947 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481349 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481352 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481355 4947 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481357 4947 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481359 4947 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481362 4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:14:16.485037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481364 4947 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481366 4947 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481369 4947 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481372 4947 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481375 4947 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481377 4947 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481380 4947 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481382 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481385 4947 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481388 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481390 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481393 4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481395 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481399 4947 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481403 4947 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481406 4947 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481408 4947 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481411 4947 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481414 4947 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:14:16.485535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481417 4947 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481419 4947 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481422 4947 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481424 4947 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481427 4947 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481429 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481432 4947 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481434 4947 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481437 4947 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481439 4947 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481442 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481445 4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481447 4947 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481450 4947 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481453 4947 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481456 4947 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481458 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481461 4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481464 4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481466 4947 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:14:16.486072 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481469 4947 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481472 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481474 4947 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481477 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481479 4947 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481481 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481484 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481486 4947 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481489 4947 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481491 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481494 4947 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481496 4947 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481499 4947 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481501 4947 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481504 4947 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481506 4947 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481509 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481511 4947 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481514 4947 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:14:16.486795 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481516 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481519 4947 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481522 4947 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481524 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481526 4947 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481529 4947 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481532 4947 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481534 4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481537 4947 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481539 4947 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481541 4947 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481544 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481547 4947 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481549 4947 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481552 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481556 4947 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481558 4947 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481561 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481564 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481567 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:14:16.487363 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481570 4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481572 4947 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481575 4947 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481577 4947 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481579 4947 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481582 4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.481584 4947 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481678 4947 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481685 4947 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481692 4947 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481710 4947 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481715 4947 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481718 4947 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481722 4947 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481726 4947 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481729 4947 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481732 4947 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481735 4947 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481739 4947 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481742 4947 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481745 4947 flags.go:64] FLAG: --cgroup-root="" Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481748 4947 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481750 4947 flags.go:64] FLAG: --client-ca-file="" Nov 03 16:14:16.487968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481753 4947 flags.go:64] FLAG: --cloud-provider="external" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481757 4947 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481761 4947 flags.go:64] FLAG: --cluster-domain="" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481764 4947 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481768 4947 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481771 4947 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481775 4947 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481779 4947 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481782 4947 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481785 4947 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481788 4947 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481791 4947 flags.go:64] FLAG: --contention-profiling="false" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481794 4947 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481797 4947 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481800 4947 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481802 4947 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481807 4947 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481810 4947 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481813 4947 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481816 4947 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481819 4947 flags.go:64] FLAG: --enable-server="true" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481822 4947 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481831 4947 flags.go:64] FLAG: --event-burst="100" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481834 4947 flags.go:64] FLAG: --event-qps="50" Nov 03 16:14:16.488575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481837 4947 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481840 4947 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481843 4947 flags.go:64] FLAG: --eviction-hard="" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481848 4947 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481851 4947 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481854 4947 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481857 4947 flags.go:64] FLAG: --eviction-soft="" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481860 4947 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481863 4947 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481866 4947 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481869 4947 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481873 4947 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481877 4947 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481880 4947 flags.go:64] FLAG: --feature-gates="" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481884 4947 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481887 4947 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481890 4947 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481893 4947 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481896 4947 flags.go:64] FLAG: --healthz-port="10248" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481899 4947 flags.go:64] FLAG: --help="false" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481902 4947 flags.go:64] FLAG: --hostname-override="ip-10-0-143-64.ec2.internal" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481905 4947 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481908 4947 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 16:14:16.489246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481911 4947 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481914 4947 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481917 4947 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481920 4947 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481923 4947 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481926 4947 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481929 4947 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481933 4947 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481936 4947 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481939 4947 flags.go:64] FLAG: --kube-reserved="" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481942 4947 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481945 4947 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481948 4947 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481951 4947 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481954 4947 flags.go:64] FLAG: --lock-file="" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481957 4947 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481960 4947 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481963 4947 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481968 4947 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481971 4947 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481974 4947 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481978 4947 flags.go:64] FLAG: --logging-format="text" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481981 4947 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481985 4947 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 16:14:16.489878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481987 4947 flags.go:64] FLAG: --manifest-url="" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481991 4947 flags.go:64] FLAG: --manifest-url-header="" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481995 4947 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.481998 4947 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482001 4947 flags.go:64] FLAG: --max-pods="110" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482004 4947 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482007 4947 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482010 4947 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482013 4947 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482016 4947 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482019 4947 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482022 4947 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482028 4947 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482032 4947 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482035 4947 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482039 4947 flags.go:64] FLAG: --pod-cidr="" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482042 4947 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482045 4947 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482048 4947 flags.go:64] FLAG: --pods-per-core="0" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482051 4947 flags.go:64] FLAG: --port="10250" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482054 4947 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482057 4947 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ed36073cce201fb9" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482060 4947 flags.go:64] FLAG: --qos-reserved="" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482063 4947 flags.go:64] FLAG: --read-only-port="10255" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482066 4947 flags.go:64] FLAG: --register-node="true" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482069 4947 flags.go:64] FLAG: --register-with-taints="" Nov 03 16:14:16.490482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482073 4947 flags.go:64] FLAG: --registry-burst="10" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482075 4947 flags.go:64] FLAG: --registry-qps="5" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482078 4947 flags.go:64] FLAG: --reserved-cpus="" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482083 4947 flags.go:64] FLAG: --reserved-memory="" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482087 4947 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482090 4947 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482093 4947 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482096 4947 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482099 4947 flags.go:64] FLAG: --runonce="false" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482102 4947 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482105 4947 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482107 4947 flags.go:64] FLAG: --seccomp-default="false" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482110 4947 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482113 4947 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482116 4947 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482120 4947 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482122 4947 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482125 4947 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482128 4947 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482131 4947 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482134 4947 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482139 4947 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482142 4947 flags.go:64] FLAG: --system-cgroups="" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482145 4947 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482150 4947 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 16:14:16.491130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482153 4947 flags.go:64] FLAG: --tls-cert-file="" Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482156 4947 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482160 4947 flags.go:64] FLAG: --tls-min-version="" Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482163 4947 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482166 4947 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482169 4947 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482172 4947 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482175 4947 flags.go:64] FLAG: --v="2" Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482180 4947 flags.go:64] FLAG: --version="false" Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482183 4947 flags.go:64] FLAG: --vmodule="" Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482189 4947 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482193 4947 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482335 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482339 4947 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482342 4947 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482345 4947 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482349 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482353 4947 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482356 4947 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482359 4947 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482362 4947 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:14:16.492165 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482365 4947 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482369 4947 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482371 4947 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482374 4947 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482377 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482380 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482383 4947 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482388 4947 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482391 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482393 4947 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482396 4947 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482398 4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482401 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482403 4947 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482406 4947 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482408 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482411 4947 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482413 4947 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482416 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:14:16.492873 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482418 4947 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482421 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482425 4947 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482427 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482431 4947 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482434 4947 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482436 4947 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482439 4947 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482441 4947 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482444 4947 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482447 4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482449 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482451 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482454 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482456 4947 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482459 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482461 4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482464 4947 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482468 4947 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482471 4947 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:14:16.493528 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482475 4947 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482478 4947 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482481 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482484 4947 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482486 4947 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482488 4947 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482491 4947 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482494 4947 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482496 4947 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482499 4947 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482501 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482504 4947 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482507 4947 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482509 4947 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482513 4947 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482516 4947 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482518 4947 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482522 4947 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482524 4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482527 4947 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:14:16.494228 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482529 4947 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482532 4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482534 4947 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482537 4947 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482540 4947 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482542 4947 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482545 4947 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482547 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482549 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482552 4947 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482554 4947 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482557 4947 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482561 4947 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482563 4947 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482566 4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482568 4947 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482571 4947 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482573 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482576 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:14:16.494923 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482578 4947 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:14:16.495476 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482581 4947 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:14:16.495476 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482583 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:14:16.495476 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482586 4947 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:14:16.495476 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482588 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:14:16.495476 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482591 4947 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:14:16.495476 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482593 4947 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:14:16.495476 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482597 4947 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:14:16.495476 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482600 4947 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:14:16.495476 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482602 4947 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:14:16.495476 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482605 4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:14:16.495476 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482608 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:14:16.495476 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482611 4947 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:14:16.495476 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482614 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:14:16.495476 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482616 4947 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:14:16.495476 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482619 4947 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:14:16.495476 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482621 4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:14:16.495476 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482625 4947 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:14:16.495476 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482627 4947 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:14:16.496091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.482653 4947 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:14:16.496091 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482689 4947 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:14:16.496091 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482711 4947 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:14:16.496091 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482715 4947 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:14:16.496091 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482717 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:14:16.496091 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482721 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:14:16.496091 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482725 4947 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:14:16.496091 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482727 4947 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:14:16.496091 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482730 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:14:16.496091 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482733 4947 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:14:16.496091 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482735 4947 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:14:16.496091 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482738 4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:14:16.496091 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482740 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:14:16.496091 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482743 4947 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:14:16.496091 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482745 4947 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:14:16.496091 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482748 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482750 4947 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482753 4947 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482755 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482758 4947 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482760 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482764 4947 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482767 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482770 4947 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482773 4947 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482776 4947 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482779 4947 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482781 4947 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482784 4947 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482787 4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482789 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482792 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482794 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482796 4947 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482799 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:14:16.496602 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482801 4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482804 4947 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482807 4947 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482812 4947 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482816 4947 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482818 4947 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482821 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482823 4947 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482826 4947 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482828 4947 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482831 4947 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482833 4947 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482836 4947 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482838 4947 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482841 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482843 4947 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482846 4947 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482848 4947 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482852 4947 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482855 4947 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:14:16.497343 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482858 4947 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482860 4947 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482863 4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482866 4947 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482869 4947 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482871 4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482874 4947 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482877 4947 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482879 4947 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482882 4947 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482885 4947 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482888 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482891 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482893 4947 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482896 4947 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482900 4947 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482902 4947 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482904 4947 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482907 4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:14:16.497939 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482909 4947 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482912 4947 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482914 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482917 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482919 4947 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482922 4947 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482925 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482927 4947 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482930 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482932 4947 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482934 4947 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482938 4947 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482941 4947 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482944 4947 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482946 4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482949 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482951 4947 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482954 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482957 4947 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482959 4947 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:14:16.498418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482962 4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:14:16.498931 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482965 4947 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:14:16.498931 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482967 4947 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:14:16.498931 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482970 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:14:16.498931 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482972 4947 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:14:16.498931 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482975 4947 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:14:16.498931 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482977 4947 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:14:16.498931 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482980 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:14:16.498931 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482985 4947 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:14:16.498931 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482989 4947 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:14:16.498931 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482992 4947 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:14:16.498931 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482995 4947 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:14:16.498931 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.482998 4947 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:14:16.498931 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.487191 4947 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 16:14:16.498931 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.487207 4947 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 16:14:16.498931 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.487225 4947 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 16:14:16.498931 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.487230 4947 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 16:14:16.498931 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487287 4947 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:14:16.498931 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487292 4947 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487295 4947 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487298 4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487301 4947 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487303 4947 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487306 4947 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487308 4947 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487311 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487314 4947 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487316 4947 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487319 4947 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487322 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487324 4947 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487327 4947 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487330 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487333 4947 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487336 4947 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487338 4947 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487341 4947 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:14:16.499372 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487343 4947 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487346 4947 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487349 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487352 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487354 4947 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487357 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487359 4947 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487362 4947 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487366 4947 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487371 4947 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487375 4947 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487379 4947 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487383 4947 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487389 4947 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487392 4947 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487396 4947 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487401 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487405 4947 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487409 4947 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:14:16.499927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487412 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487416 4947 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487420 4947 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487424 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487428 4947 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487432 4947 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487436 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487440 4947 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487443 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487448 4947 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487451 4947 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487456 4947 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487460 4947 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487464 4947 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487468 4947 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487472 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487477 4947 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487481 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487486 4947 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:14:16.500541 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487490 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487494 4947 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487499 4947 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487506 4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487512 4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487516 4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487520 4947 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487524 4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487528 4947 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487532 4947 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487536 4947 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487540 4947 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487544 4947 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487547 4947 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487552 4947 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487555 4947 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487559 4947 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487563 4947 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487567 4947 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487571 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:14:16.501041 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487575 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487579 4947 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487583 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487587 4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487591 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487595 4947 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487599 4947 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487604 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487608 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487612 4947 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487617 4947 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487621 4947 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487627 4947 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487634 4947 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487639 4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487643 4947 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487648 4947 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487653 4947 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487658 4947 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:14:16.501535 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487662 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:14:16.502020 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487667 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:14:16.502020 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487671 4947 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:14:16.502020 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487676 4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:14:16.502020 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487680 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:14:16.502020 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487684 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:14:16.502020 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487689 4947 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:14:16.502020 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487718 4947 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:14:16.502020 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487724 4947 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:14:16.502020 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.487772 4947 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:14:16.502020 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.487995 4947 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:14:16.502020 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488003 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:14:16.502020 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488007 4947 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:14:16.502020 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488012 4947 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:14:16.502020 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488016 4947 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:14:16.502020 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488022 4947 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:14:16.502020 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488026 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488030 4947 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488034 4947 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488038 4947 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488042 4947 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488046 4947 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488049 4947 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488053 4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488057 4947 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488061 4947 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488064 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488068 4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488072 4947 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488076 4947 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488080 4947 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488085 4947 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488089 4947 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488092 4947 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488098 4947 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:14:16.502501 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488104 4947 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488108 4947 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488112 4947 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488117 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488121 4947 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488124 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488128 4947 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488132 4947 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488136 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488140 4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488144 4947 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488148 4947 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488152 4947 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488156 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488159 4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488163 4947 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488167 4947 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488171 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488175 4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488179 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:14:16.503062 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488184 4947 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488188 4947 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488192 4947 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488196 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488200 4947 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488204 4947 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488208 4947 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488212 4947 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488215 4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488220 4947 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488223 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488226 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488228 4947 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488231 4947 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488233 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488237 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488240 4947 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488242 4947 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488245 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488247 4947 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:14:16.503745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488250 4947 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488253 4947 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488255 4947 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488258 4947 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488260 4947 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488263 4947 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488265 4947 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488268 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488271 4947 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488273 4947 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488276 4947 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488278 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488281 4947 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488283 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488286 4947 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488288 4947 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488290 4947 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488293 4947 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488295 4947 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488298 4947 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:14:16.504248 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488300 4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488305 4947 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488309 4947 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488312 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488315 4947 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488317 4947 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488320 4947 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488323 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488325 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488328 4947 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488330 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488333 4947 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488335 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488338 4947 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488340 4947 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488343 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488346 4947 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488348 4947 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488351 4947 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:14:16.504791 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488353 4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:14:16.505323 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488356 4947 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:14:16.505323 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.488388 4947 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 16:14:16.505323 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488426 4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 16:14:16.505323 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488430 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 16:14:16.505323 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488433 4947 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 16:14:16.505323 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488436 4947 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 16:14:16.505323 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488439 4947 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 16:14:16.505323 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488442 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 16:14:16.505323 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488444 4947 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 16:14:16.505323 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488446 4947 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 16:14:16.505323 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488449 4947 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 16:14:16.505323 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488451 4947 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 16:14:16.505323 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488454 4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 16:14:16.505323 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488456 4947 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 16:14:16.505323 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488458 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 16:14:16.505323 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488461 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488464 4947 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488466 4947 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488468 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488471 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488474 4947 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488476 4947 feature_gate.go:386] unrecognized feature gate: Example Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488479 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488481 4947 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488483 4947 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488486 4947 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488488 4947 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488491 4947 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488493 4947 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488495 4947 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488498 4947 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488500 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488502 4947 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488505 4947 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 16:14:16.505768 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488507 4947 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488510 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488512 4947 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488515 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488517 4947 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488519 4947 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488522 4947 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488524 4947 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488526 4947 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488529 4947 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488531 4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488535 4947 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488539 4947 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488541 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488550 4947 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488554 4947 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488557 4947 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488560 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488563 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 16:14:16.506235 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488565 4947 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488568 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488570 4947 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488573 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488575 4947 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488578 4947 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488581 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488583 4947 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488586 4947 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488589 4947 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488592 4947 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488594 4947 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488597 4947 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488599 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488602 4947 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488606 4947 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488608 4947 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488611 4947 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488613 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488615 4947 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 16:14:16.506707 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488618 4947 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488620 4947 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488623 4947 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488625 4947 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488627 4947 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488630 4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488632 4947 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488635 4947 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488638 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488641 4947 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488649 4947 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488652 4947 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488655 4947 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488666 4947 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488669 4947 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488672 4947 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488676 4947 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488679 4947 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488682 4947 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 16:14:16.507252 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488685 4947 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488688 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488690 4947 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488712 4947 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488717 4947 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488721 4947 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488723 4947 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488726 4947 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488729 4947 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488732 4947 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488734 4947 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488737 4947 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488739 4947 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488742 4947 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488746 4947 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:16.488748 4947 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.488808 4947 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.489862 4947 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.489912 4947 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 16:14:16.507760 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.490088 4947 server.go:1014] "Starting client certificate rotation" Nov 03 16:14:16.508232 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.490230 4947 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 16:14:16.508232 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.490272 4947 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 16:08:36 +0000 UTC" deadline="2028-07-23 17:31:26.103888446 +0000 UTC" Nov 03 16:14:16.508232 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.490300 4947 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15073h17m9.613590309s" Nov 03 16:14:16.508232 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.490520 4947 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:14:16.508232 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.493492 4947 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 16:14:16.508232 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.494797 4947 log.go:25] "Validated CRI v1 runtime API" Nov 03 16:14:16.508232 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.496526 4947 log.go:25] "Validated CRI v1 image API" Nov 03 16:14:16.508232 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.497237 4947 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 16:14:16.508232 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.499231 4947 fs.go:135] Filesystem UUIDs: map[2ce85333-6709-41c1-8952-c530e2aa739b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ecd882ce-6f3b-4a6d-9895-209189573d62:/dev/nvme0n1p4] Nov 03 16:14:16.508608 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.499252 4947 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/021004413f8c8b747965882c61d802c3df951d7f8ad43dc3f87794731f7d84c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/021004413f8c8b747965882c61d802c3df951d7f8ad43dc3f87794731f7d84c9/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1baf87fe253f97ace50fbc9aa73885717a8e523863b6a5af5a0c209c84a6226a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1baf87fe253f97ace50fbc9aa73885717a8e523863b6a5af5a0c209c84a6226a/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ba6d971399525cdd8e1815130a771c92ed0d3452825a191e0dadf1428bd25ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ba6d971399525cdd8e1815130a771c92ed0d3452825a191e0dadf1428bd25ff/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c6d7e35341f617447534ea331414166e75c5c6220a636690c5e2db938cc4ad7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c6d7e35341f617447534ea331414166e75c5c6220a636690c5e2db938cc4ad7/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/324ac350996ad1f2ce0d43d06952e2852e07aabc164151e5266c319e5df349c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/324ac350996ad1f2ce0d43d06952e2852e07aabc164151e5266c319e5df349c4/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33063db3a0abdf281cc0d58a01ecee734a14c03e79a2cd030a1be0adc5f398d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33063db3a0abdf281cc0d58a01ecee734a14c03e79a2cd030a1be0adc5f398d7/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33f6cb62246e179f45746f693ebceda4e77fdf4ccc88bf16da7c137ce2d02e70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33f6cb62246e179f45746f693ebceda4e77fdf4ccc88bf16da7c137ce2d02e70/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/385883e4ba88316132665e384730bd687c493c1006cfdbcc39d7ac1b870ed4bf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/385883e4ba88316132665e384730bd687c493c1006cfdbcc39d7ac1b870ed4bf/userdata/shm major:0 minor:304 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3938c3ec0d18e946ab09733a2e7c128bec132fe52206ebc22737c77e423812a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3938c3ec0d18e946ab09733a2e7c128bec132fe52206ebc22737c77e423812a1/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b29f2877c3d7d4b0259887e73ffe8e7d4895d8ebae92ff51273eb85de140587/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b29f2877c3d7d4b0259887e73ffe8e7d4895d8ebae92ff51273eb85de140587/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c7564eb1c2844cff816a627465bb36ce2d4260057a1b479a34de3a577540c15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c7564eb1c2844cff816a627465bb36ce2d4260057a1b479a34de3a577540c15/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60a1c67ee846d8eb71f014ba0b3afff8b717a1c232f4e2abacc8bd352df6038d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60a1c67ee846d8eb71f014ba0b3afff8b717a1c232f4e2abacc8bd352df6038d/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/60a6fb6a35818963c1f0b23463654b6788fe8550e94e88e2690ab09d530769ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/60a6fb6a35818963c1f0b23463654b6788fe8550e94e88e2690ab09d530769ac/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61b2a96cbbca5b98e23a41a60272db802884d84506232d4fa833e853770ebe30/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61b2a96cbbca5b98e23a41a60272db802884d84506232d4fa833e853770ebe30/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73372b2c138ebb0b5a2ce8af6592946e65bd17ddcd56ecbb1b40093dc8c02fea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73372b2c138ebb0b5a2ce8af6592946e65bd17ddcd56ecbb1b40093dc8c02fea/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d99312e03eaf44c940fea1c6bd9e48ab0f5fdf296cf48a30b96f996e257b1c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d99312e03eaf44c940fea1c6bd9e48ab0f5fdf296cf48a30b96f996e257b1c9/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e34181ab541ebf6938d0e45793de04f3c6539adb7b03bedc6ca812ab2699f97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e34181ab541ebf6938d0e45793de04f3c6539adb7b03bedc6ca812ab2699f97/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f8c99e401fa37c70cbaa90bad16117ee66798b83394f66d5c63cd769043fb74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f8c99e401fa37c70cbaa90bad16117ee66798b83394f66d5c63cd769043fb74/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b20754d576e2fb6d6e22e148d116e2c252bd0e552948343d86cdd6ff56741aca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b20754d576e2fb6d6e22e148d116e2c252bd0e552948343d86cdd6ff56741aca/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc768d9eab330521ad346bc708b210ee09b31e0192bcaf47f0532553dfd2f619/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc768d9eab330521ad346bc708b210ee09b31e0192bcaf47f0532553dfd2f619/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0d5f41f36ea2f8106c2bd9f0ce12c720e987eddbe6cde2bbc673bcd2a235f67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0d5f41f36ea2f8106c2bd9f0ce12c720e987eddbe6cde2bbc673bcd2a235f67/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cd11594e79cefb0bb847b42f8639cfaf6757665400c707c490511b666e78a220/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cd11594e79cefb0bb847b42f8639cfaf6757665400c707c490511b666e78a220/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da3d72656b51ca863a1d7ed72013624774cd07b382ce9eaf834827431b293899/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da3d72656b51ca863a1d7ed72013624774cd07b382ce9eaf834827431b293899/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03cc593d-32bc-4453-8a45-2b546f862acf/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/03cc593d-32bc-4453-8a45-2b546f862acf/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03cc593d-32bc-4453-8a45-2b546f862acf/volumes/kubernetes.io~projected/kube-api-access-rl4vw:{mountpoint:/var/lib/kubelet/pods/03cc593d-32bc-4453-8a45-2b546f862acf/volumes/kubernetes.io~projected/kube-api-access-rl4vw major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03cc593d-32bc-4453-8a45-2b546f862acf/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/03cc593d-32bc-4453-8a45-2b546f862acf/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03cc593d-32bc-4453-8a45-2b546f862acf/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/03cc593d-32bc-4453-8a45-2b546f862acf/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/110bff76-1581-4379-8468-441a3a81dea2/volumes/kubernetes.io~projected/kube-api-access-l9slx:{mountpoint:/var/lib/kubelet/pods/110bff76-1581-4379-8468-441a3a81dea2/volumes/kubernetes.io~projected/kube-api-access-l9slx major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1566c753-13b1-4495-9268-05e3676d82bd/volumes/kubernetes.io~projected/kube-api-access-vwcf4:{mountpoint:/var/lib/kubelet/pods/1566c753-13b1-4495-9268-05e3676d82bd/volumes/kubernetes.io~projected/kube-api-access-vwcf4 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1580792e-eb86-4d79-92a7-8055e6a8aaff/volumes/kubernetes.io~projected/kube-api-access-8fx4n:{mountpoint:/var/lib/kubelet/pods/1580792e-eb86-4d79-92a7-8055e6a8aaff/volumes/kubernetes.io~projected/kube-api-access-8fx4n major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1580792e-eb86-4d79-92a7-8055e6a8aaff/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1580792e-eb86-4d79-92a7-8055e6a8aaff/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/216c0d66-a3ec-4748-b761-3420536bfa3a/volumes/kubernetes.io~projected/kube-api-access-wz7ht:{mountpoint:/var/lib/kubelet/pods/216c0d66-a3ec-4748-b761-3420536bfa3a/volumes/kubernetes.io~projected/kube-api-access-wz7ht major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/216c0d66-a3ec-4748-b761-3420536bfa3a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/216c0d66-a3ec-4748-b761-3420536bfa3a/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cbd4c91-69f0-47be-94be-e451d44b4992/volumes/kubernetes.io~projected/kube-api-access-l4h7b:{mountpoint:/var/lib/kubelet/pods/3cbd4c91-69f0-47be-94be-e451d44b4992/volumes/kubernetes.io~projected/kube-api-access-l4h7b major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4b370319-6de8-41e2-ae5f-21267a2fba74/volumes/kubernetes.io~projected/kube-api-access-pzhxd:{mountpoint:/var/lib/kubelet/pods/4b370319-6de8-41e2-ae5f-21267a2fba74/volumes/kubernetes.io~projected/kube-api-access-pzhxd major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4cbc83f2-64b1-4804-a180-43df6459ffa4/volumes/kubernetes.io~projected/kube-api-access-42fxm:{mountpoint:/var/lib/kubelet/pods/4cbc83f2-64b1-4804-a180-43df6459ffa4/volumes/kubernetes.io~projected/kube-api-access-42fxm major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55767581-2059-49bf-8dd0-69e31d5adb83/volumes/kubernetes.io~projected/kube-api-access-2z647:{mountpoint:/var/lib/kubelet/pods/55767581-2059-49bf-8dd0-69e31d5adb83/volumes/kubernetes.io~projected/kube-api-access-2z647 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ced7a93-d9f0-44de-9a76-34a84e41843b/volumes/kubernetes.io~projected/kube-api-access-przlh:{mountpoint:/var/lib/kubelet/pods/5ced7a93-d9f0-44de-9a76-34a84e41843b/volumes/kubernetes.io~projected/kube-api-access-przlh major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66b43213-9ec2-4164-9f8d-d7621bd57eb3/volumes/kubernetes.io~projected/kube-api-access-75xq7:{mountpoint:/var/lib/kubelet/pods/66b43213-9ec2-4164-9f8d-d7621bd57eb3/volumes/kubernetes.io~projected/kube-api-access-75xq7 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66b43213-9ec2-4164-9f8d-d7621bd57eb3/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/66b43213-9ec2-4164-9f8d-d7621bd57eb3/volumes/kubernetes.io~secret/default-certificate major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66b43213-9ec2-4164-9f8d-d7621bd57eb3/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/66b43213-9ec2-4164-9f8d-d7621bd57eb3/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7731b228-5421-4121-9228-1f19a850ef9a/volumes/kubernetes.io~projected/kube-api-access-vp88d:{mountpoint:/var/lib/kubelet/pods/7731b228-5421-4121-9228-1f19a850ef9a/volumes/kubernetes.io~projected/kube-api-access-vp88d major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78e9f545-8400-48b6-968d-eb65dc94b471/volumes/kubernetes.io~projected/kube-api-access-9g2jb:{mountpoint:/var/lib/kubelet/pods/78e9f545-8400-48b6-968d-eb65dc94b471/volumes/kubernetes.io~projected/kube-api-access-9g2jb major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ec02fd2-78a5-4813-b6e4-4edefa217fe1/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7ec02fd2-78a5-4813-b6e4-4edefa217fe1/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ec02fd2-78a5-4813-b6e4-4edefa217fe1/volumes/kubernetes.io~projected/kube-api-access-cfv4w:{mountpoint:/var/lib/kubelet/pods/7ec02fd2-78a5-4813-b6e4-4edefa217fe1/volumes/kubernetes.io~projected/kube-api-access-cfv4w major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ec02fd2-78a5-4813-b6e4-4edefa217fe1/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7ec02fd2-78a5-4813-b6e4-4edefa217fe1/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/812067ca-eef5-4ab1-a24e-61796ac0721b/volumes/kubernetes.io~projected/kube-api-access-hn2hk:{mountpoint:/var/lib/kubelet/pods/812067ca-eef5-4ab1-a24e-61796ac0721b/volumes/kubernetes.io~projected/kube-api-access-hn2hk major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c96f345-2cda-4554-bef3-be416dc50225/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8c96f345-2cda-4554-bef3-be416dc50225/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f85dfa8-9158-437d-b151-d8eb8fe65b7b/volumes/kubernetes.io~projected/kube-api-access-xdzb4:{mountpoint:/var/lib/kubelet/pods/8f85dfa8-9158-437d-b151-d8eb8fe65b7b/volumes/kubernetes.io~projected/kube-api-access-xdzb4 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f85dfa8-9158-437d-b151-d8eb8fe65b7b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8f85dfa8-9158-437d-b151-d8eb8fe65b7b/volumes/kubernetes.io~secret/cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94fb68fc-92af-4e72-824b-7092f9875560/volumes/kubernetes.io~projected/kube-api-access-2d2cr:{mountpoint:/var/lib/kubelet/pods/94fb68fc-92af-4e72-824b-7092f9875560/volumes/kubernetes.io~projected/kube-api-access-2d2cr major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a13e4eb2-b4f2-4f66-85d3-8bedb078823c/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/a13e4eb2-b4f2-4f66-85d3-8bedb078823c/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a13e4eb2-b4f2-4f66-85d3-8bedb078823c/volumes/kubernetes.io~projected/kube-api-access-z4dbx:{mountpoint:/var/lib/kubelet/pods/a13e4eb2-b4f2-4f66-85d3-8bedb078823c/volumes/kubernetes.io~projected/kube-api-access-z4dbx major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a13e4eb2-b4f2-4f66-85d3-8bedb078823c/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/a13e4eb2-b4f2-4f66-85d3-8bedb078823c/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a13e4eb2-b4f2-4f66-85d3-8bedb078823c/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/a13e4eb2-b4f2-4f66-85d3-8bedb078823c/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3ce380e-3ddd-42bd-867c-9bba84f2bab4/volumes/kubernetes.io~projected/kube-api-access-gjwvk:{mountpoint:/var/lib/kubelet/pods/a3ce380e-3ddd-42bd-867c-9bba84f2bab4/volumes/kubernetes.io~projected/kube-api-access-gjwvk major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7f5be2e-c57e-432f-a5bb-144709f3b492/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a7f5be2e-c57e-432f-a5bb-144709f3b492/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7f5be2e-c57e-432f-a5bb-144709f3b492/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a7f5be2e-c57e-432f-a5bb-144709f3b492/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7f5be2e-c57e-432f-a5bb-144709f3b492/volumes/kubernetes.io~projected/kube-api-access-j9hhm:{mountpoint:/var/lib/kubelet/pods/a7f5be2e-c57e-432f-a5bb-144709f3b492/volumes/kubernetes.io~projected/kube-api-access-j9hhm major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9c106a2-ce74-4c57-80c1-b1f75da0d43d/volumes/kubernetes.io~projected/kube-api-access-4rznq:{mountpoint:/var/lib/kubelet/pods/b9c106a2-ce74-4c57-80c1-b1f75da0d43d/volumes/kubernetes.io~projected/kube-api-access-4rznq major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9c106a2-ce74-4c57-80c1-b1f75da0d43d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b9c106a2-ce74-4c57-80c1-b1f75da0d43d/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba6c03a7-1ba9-48c0-b67b-6fb26263425e/volumes/kubernetes.io~projected/kube-api-access-tszlg:{mountpoint:/var/lib/kubelet/pods/ba6c03a7-1ba9-48c0-b67b-6fb26263425e/volumes/kubernetes.io~projected/kube-api-access-tszlg major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be60fb17-308e-420d-b9e0-d9a05c0c752f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/be60fb17-308e-420d-b9e0-d9a05c0c752f/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8c316f-51ce-4c69-865f-1d481ff30630/volumes/kubernetes.io~projected/kube-api-access-gtbz7:{mountpoint:/var/lib/kubelet/pods/bf8c316f-51ce-4c69-865f-1d481ff30630/volumes/kubernetes.io~projected/kube-api-access-gtbz7 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bf8c316f-51ce-4c69-865f-1d481ff30630/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/bf8c316f-51ce-4c69-865f-1d481ff30630/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2ded168-1814-4922-9eee-00caf304b8af/volumes/kubernetes.io~projected/kube-api-access-svnps:{mountpoint:/var/lib/kubelet/pods/c2ded168-1814-4922-9eee-00caf304b8af/volumes/kubernetes.io~projected/kube-api-access-svnps major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28/volumes/kubernetes.io~projected/kube-api-access-phmbx:{mountpoint:/var/lib/kubelet/pods/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28/volumes/kubernetes.io~projected/kube-api-access-phmbx major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd/volumes/kubernetes.io~projected/kube-api-access-5jjxv:{mountpoint:/var/lib/kubelet/pods/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd/volumes/kubernetes.io~projected/kube-api-access-5jjxv major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f02c434e-ede0-4bea-8830-f3444c21a6a6/volumes/kubernetes.io~projected/kube-api-access-kfkpb:{mountpoint:/var/lib/kubelet/pods/f02c434e-ede0-4bea-8830-f3444c21a6a6/volumes/kubernetes.io~projected/kube-api-access-kfkpb major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa01a744-9121-4f6a-b280-dba215c1ab8f/volumes/kubernetes.io~projected/kube-api-access-d4vdm:{mountpoint:/var/lib/kubelet/pods/fa01a744-9121-4f6a-b280-dba215c1ab8f/volumes/kubernetes.io~projected/kube-api-access-d4vdm major:0 minor:223 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/00ac46a444834c09ec69acf30de80acbc0b139eb064433e5d45094aa4cb9ed9c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/08b512727340343b26c843c9ab197a91798f93109d2b6e6d85901a55e8b017fb/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/87553675c1db189adcecb0f6bd5dfadb03a993dd99d6bf35c065f37cf9b808e1/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/099a7041032bb3c2eb2e8f9c3c939442cdda990fccb94fec5f5ed25087262340/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/f56f0f54d112c5731d6c7a4a56dbfc1892610335482e48953f9a62ef69f84f57/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/1b67e437c1cb0ce9b687c64ddc9f7e63f62d99ac4e0ab5f56bf0e3dc9edfd79e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/34b9ffc5f652dae2f4958d9c8cd0c34935af167ae6cf9c8337870d257775bf67/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/927f891a3fd05febef0ff9cb095a1095ad702bbfc4d71ed9f3ef9db63c604700/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/48bc5eb13a3f038926df9e68617acf741a36911acac40e5a9f106b4c2de3a4f9/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/a8614b66574ca502b80a64cfa885c219d186c937e1a04072487cf8798a360fc9/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/9123a6ee53a153db6bbf645ea213912bf7839c84962fb4fcbe3b928cfcb3db10/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/1c0f0f2f58c0d01119e53616e51e5c01dd44efe67dbc82796fb7e2f20f5162d1/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/3dc07f2d06b0278391724e1d832c7e52ee4ed70a40c259d2790da1cee755be4c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5955eaf78dd4b49507529252ab2deecf854c915bbe034cfde718d76939737521/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/bfd476b6a60322c765a6eb5ed89f81ce44c83ab6d709f85fd28f307ecdfdfc8a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/60374b5d898689b380c716a463392b04eff8e23d31f3fb45a29cf45c6fddd1ab/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/9e5b04a92032f1a2c161d4517b32d8f10402406f984646fb7502c137161e1467/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d6c40ae2c30b76df9cd38fde416fa6304c69bc82b7e9f4a276705d5e5c185136/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/2e8f9b78461e39c85ea280b66eab92012bf69ced11bd18c6d177e0ece26f8f30/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/794b252403d85d98d1306d31abe7ae2bbbbeee6da093fce0edd1bcd464312d69/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e418adbcfe9549c451e2e3f7af2b3284c5e3b88f1647ba6a2bdba392a5ab14ff/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/d2b91eb94b785a871cc3d80614a1f9d7a9e8f6bbe671f9eaeafe1f442854d43a/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/576ad54ccfa34e24907888e763338d5870cbcf2a7850ec213cf08355ece12923/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/f90ee12cb5c9e6812a44e004b4519dce88b75c2d7b3d4d021dea5f837adb6ab0/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/4ba48b8204f2637ebd603cd177aeed6fd2a50142409979a714dff50d7cdd5648/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/80685f8e7d8f33f8453f4f43e8bc4aa675ada5fee5d2add021666dc82e5d4c2c/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/f94eba058e4f22543761d994ef13c6caffa011bd67725cc3f9fb225652e84848/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/787a76430f06cfe5f72b3db46e24e0a79fbfb24a076891249cd4552c4e20ff91/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/1433bac658cf984eda63205e35e7ada273e5c58e21d04af525d5feffbbcfd7fb/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/a924e1dd680b3428aae94c4db13ca7e7cb9cc7936271e2a0141ad44696d02bff/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/8fcb1b49d0e6a70778bf369443117abfb0a7447fec0efcf0251a52937925e386/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/3a33de17c5b598cd4abe8d3c3fa91763cb63f20eca493b3b69ea3a1d74c351e5/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/b5414aebfe628e3a70db024a644c30126b9242ded4be8d410ab47ab2c1f160fd/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/8788845e121a0fd76c6115f9311625f84290d431be09aa573b379eb0b80fd993/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/1788d603f615838872c75f6a8b353fe6c178ab266a2caeecce8e27fdcc1ea19b/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/791113616b6a7f408e49533c73d80cc99a20ceb86142e685fc4f07614935c1b6/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/557b173c9a7cf2a127b071e16c9481764a461f1bd46614122cb9138fb93a14a3/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/cf1ee3ca5214d570a14a43ea408314016a4c6623438db889edf7855d0917a49e/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/382472ab675eafec6e010c7b65ece21a138632746a2967c6994877a48be67083/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/6b171e39263b7345ef80c319f7083320fe220eeb78a34454f37b6ead6745500a/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/fb431e72200531add8a8c0b8eada94e82778d2661a4222463e82ee221cdb1f0a/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/0397a17dd78f3fe3ead565436ce98fbf970b6bd8ac56566734cf524d7d8bd99e/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/df0160c3b632f1776a95dfa97d980e7911eccfdd4a8481dad6c0eed5e9219d32/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/d1a0c3be87a5ca37ffd7d29445ce6730a39aacdf477d4f2762c5b1c21bc0e610/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/9d934057966bccbff72f3f747c55f7ef2ab70e8b0ba4e0651395305fa0c64826/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/e8c6e81470d7b146c4df2e03bba20fe96376256cdf481c3d64dee826b0e3c9d4/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/f4bd56ceabf6416ebdf8b70d21b1fac196bc008700165998e372dbd8e681c582/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/64f65c82f69bb2cf53fdb4766b4a8b8bbc3bc2e7a271b7950f7c22d0c61da54f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/2ba3dab24a8cb72fab9a7056b1254e1a584d0753a04f18a6a865d8c8ac2090ee/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/b515d104ac348ddb320cd3d6084ac6e916389f45025cfde16ab84bafb946a13f/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/bdf3d28de20b4e0366c3f4ad4a7cc46c14d008dd22dd405d8ac88993df035ec4/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/8cd018bd9626ccaf4163243b4652dade243e250e581ffb4505c1cb483ff4f9c6/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/97f45425a0f2b8453691474600a58c4bd71617cdfe9f46838cca5687ee15bd88/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/e2d9ee9bd49fef13c979ca3c5eef3284a27c110e5179d69287545d9a4d3f48a3/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/e236b5d760d87bfb5e48b238017488ed1bbb1755a1e23856fd33e3eb35c7b639/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/62d7dead74873967e52e6cde9dbd70f488474f2517586888b973779498318a3d/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 16:14:16.508976 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.507084 4947 manager.go:217] Machine: {Timestamp:2026-11-03 16:14:16.506596224 +0000 UTC m=+0.095596185 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096448 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25b77bc0e7f44a77e44724648fcbe4 SystemUUID:ec25b77b-c0e7-f44a-77e4-4724648fcbe4 BootID:64cfe853-8223-4424-ba46-656247125e46 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/812067ca-eef5-4ab1-a24e-61796ac0721b/volumes/kubernetes.io~projected/kube-api-access-hn2hk DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7ec02fd2-78a5-4813-b6e4-4edefa217fe1/volumes/kubernetes.io~projected/kube-api-access-cfv4w DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7731b228-5421-4121-9228-1f19a850ef9a/volumes/kubernetes.io~projected/kube-api-access-vp88d DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60a6fb6a35818963c1f0b23463654b6788fe8550e94e88e2690ab09d530769ac/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8c316f-51ce-4c69-865f-1d481ff30630/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cd11594e79cefb0bb847b42f8639cfaf6757665400c707c490511b666e78a220/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1580792e-eb86-4d79-92a7-8055e6a8aaff/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c96f345-2cda-4554-bef3-be416dc50225/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:294 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/110bff76-1581-4379-8468-441a3a81dea2/volumes/kubernetes.io~projected/kube-api-access-l9slx DeviceMajor:0 DeviceMinor:288 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ec02fd2-78a5-4813-b6e4-4edefa217fe1/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a7f5be2e-c57e-432f-a5bb-144709f3b492/volumes/kubernetes.io~projected/kube-api-access-j9hhm DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2ded168-1814-4922-9eee-00caf304b8af/volumes/kubernetes.io~projected/kube-api-access-svnps DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b20754d576e2fb6d6e22e148d116e2c252bd0e552948343d86cdd6ff56741aca/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7f5be2e-c57e-432f-a5bb-144709f3b492/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4cbc83f2-64b1-4804-a180-43df6459ffa4/volumes/kubernetes.io~projected/kube-api-access-42fxm DeviceMajor:0 DeviceMinor:215 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28/volumes/kubernetes.io~projected/kube-api-access-phmbx DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a13e4eb2-b4f2-4f66-85d3-8bedb078823c/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73372b2c138ebb0b5a2ce8af6592946e65bd17ddcd56ecbb1b40093dc8c02fea/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/be60fb17-308e-420d-b9e0-d9a05c0c752f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03cc593d-32bc-4453-8a45-2b546f862acf/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a13e4eb2-b4f2-4f66-85d3-8bedb078823c/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:209 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f85dfa8-9158-437d-b151-d8eb8fe65b7b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da3d72656b51ca863a1d7ed72013624774cd07b382ce9eaf834827431b293899/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8f85dfa8-9158-437d-b151-d8eb8fe65b7b/volumes/kubernetes.io~projected/kube-api-access-xdzb4 DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/60a1c67ee846d8eb71f014ba0b3afff8b717a1c232f4e2abacc8bd352df6038d/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/324ac350996ad1f2ce0d43d06952e2852e07aabc164151e5266c319e5df349c4/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7f5be2e-c57e-432f-a5bb-144709f3b492/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/4b370319-6de8-41e2-ae5f-21267a2fba74/volumes/kubernetes.io~projected/kube-api-access-pzhxd DeviceMajor:0 DeviceMinor:214 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/03cc593d-32bc-4453-8a45-2b546f862acf/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:222 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1baf87fe253f97ace50fbc9aa73885717a8e523863b6a5af5a0c209c84a6226a/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc768d9eab330521ad346bc708b210ee09b31e0192bcaf47f0532553dfd2f619/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3ce380e-3ddd-42bd-867c-9bba84f2bab4/volumes/kubernetes.io~projected/kube-api-access-gjwvk DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d99312e03eaf44c940fea1c6bd9e48ab0f5fdf296cf48a30b96f996e257b1c9/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33f6cb62246e179f45746f693ebceda4e77fdf4ccc88bf16da7c137ce2d02e70/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0d5f41f36ea2f8106c2bd9f0ce12c720e987eddbe6cde2bbc673bcd2a235f67/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03cc593d-32bc-4453-8a45-2b546f862acf/volumes/kubernetes.io~projected/kube-api-access-rl4vw DeviceMajor:0 DeviceMinor:212 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b9c106a2-ce74-4c57-80c1-b1f75da0d43d/volumes/kubernetes.io~projected/kube-api-access-4rznq DeviceMajor:0 DeviceMinor:227 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78e9f545-8400-48b6-968d-eb65dc94b471/volumes/kubernetes.io~projected/kube-api-access-9g2jb DeviceMajor:0 DeviceMinor:210 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/216c0d66-a3ec-4748-b761-3420536bfa3a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/94fb68fc-92af-4e72-824b-7092f9875560/volumes/kubernetes.io~projected/kube-api-access-2d2cr DeviceMajor:0 DeviceMinor:335 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ba6d971399525cdd8e1815130a771c92ed0d3452825a191e0dadf1428bd25ff/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/03cc593d-32bc-4453-8a45-2b546f862acf/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/216c0d66-a3ec-4748-b761-3420536bfa3a/volumes/kubernetes.io~projected/kube-api-access-wz7ht DeviceMajor:0 DeviceMinor:220 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/021004413f8c8b747965882c61d802c3df951d7f8ad43dc3f87794731f7d84c9/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f02c434e-ede0-4bea-8830-f3444c21a6a6/volumes/kubernetes.io~projected/kube-api-access-kfkpb DeviceMajor:0 DeviceMinor:314 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3938c3ec0d18e946ab09733a2e7c128bec132fe52206ebc22737c77e423812a1/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/55767581-2059-49bf-8dd0-69e31d5adb83/volumes/kubernetes.io~projected/kube-api-access-2z647 DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1580792e-eb86-4d79-92a7-8055e6a8aaff/volumes/kubernetes.io~projected/kube-api-access-8fx4n DeviceMajor:0 DeviceMinor:211 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66b43213-9ec2-4164-9f8d-d7621bd57eb3/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:206 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ba6c03a7-1ba9-48c0-b67b-6fb26263425e/volumes/kubernetes.io~projected/kube-api-access-tszlg DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61b2a96cbbca5b98e23a41a60272db802884d84506232d4fa833e853770ebe30/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1566c753-13b1-4495-9268-05e3676d82bd/volumes/kubernetes.io~projected/kube-api-access-vwcf4 DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f8c99e401fa37c70cbaa90bad16117ee66798b83394f66d5c63cd769043fb74/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66b43213-9ec2-4164-9f8d-d7621bd57eb3/volumes/kubernetes.io~projected/kube-api-access-75xq7 DeviceMajor:0 DeviceMinor:221 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c7564eb1c2844cff816a627465bb36ce2d4260057a1b479a34de3a577540c15/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bf8c316f-51ce-4c69-865f-1d481ff30630/volumes/kubernetes.io~projected/kube-api-access-gtbz7 DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33063db3a0abdf281cc0d58a01ecee734a14c03e79a2cd030a1be0adc5f398d7/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c6d7e35341f617447534ea331414166e75c5c6220a636690c5e2db938cc4ad7/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/66b43213-9ec2-4164-9f8d-d7621bd57eb3/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3cbd4c91-69f0-47be-94be-e451d44b4992/volumes/kubernetes.io~projected/kube-api-access-l4h7b DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a13e4eb2-b4f2-4f66-85d3-8bedb078823c/volumes/kubernetes.io~projected/kube-api-access-z4dbx DeviceMajor:0 DeviceMinor:216 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fa01a744-9121-4f6a-b280-dba215c1ab8f/volumes/kubernetes.io~projected/kube-api-access-d4vdm DeviceMajor:0 DeviceMinor:223 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ec02fd2-78a5-4813-b6e4-4edefa217fe1/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd/volumes/kubernetes.io~projected/kube-api-access-5jjxv DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b9c106a2-ce74-4c57-80c1-b1f75da0d43d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5ced7a93-d9f0-44de-9a76-34a84e41843b/volumes/kubernetes.io~projected/kube-api-access-przlh DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a13e4eb2-b4f2-4f66-85d3-8bedb078823c/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e34181ab541ebf6938d0e45793de04f3c6539adb7b03bedc6ca812ab2699f97/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b29f2877c3d7d4b0259887e73ffe8e7d4895d8ebae92ff51273eb85de140587/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/385883e4ba88316132665e384730bd687c493c1006cfdbcc39d7ac1b870ed4bf/userdata/shm DeviceMajor:0 DeviceMinor:304 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:021004413f8c8b7 MacAddress:9e:2b:d7:c8:b0:22 Speed:10000 Mtu:8901} {Name:1baf87fe253f97a MacAddress:de:1e:43:b1:05:71 Speed:10000 Mtu:8901} {Name:324ac350996ad1f MacAddress:92:ad:a8:6a:be:69 Speed:10000 Mtu:8901} {Name:33063db3a0abdf2 MacAddress:76:23:50:c7:11:3b Speed:10000 Mtu:8901} {Name:385883e4ba88316 MacAddress:be:48:e0:36:27:08 Speed:10000 Mtu:8901} {Name:3938c3ec0d18e94 MacAddress:86:19:34:3e:e0:96 Speed:10000 Mtu:8901} {Name:4b29f2877c3d7d4 MacAddress:62:59:a8:1c:fe:d5 Speed:10000 Mtu:8901} {Name:4c7564eb1c2844c MacAddress:26:d2:ef:5b:4e:0e Speed:10000 Mtu:8901} {Name:61b2a96cbbca5b9 MacAddress:d6:52:6d:36:28:8f Speed:10000 Mtu:8901} {Name:73372b2c138ebb0 MacAddress:aa:73:82:b3:92:39 Speed:10000 Mtu:8901} {Name:9e34181ab541ebf MacAddress:66:09:29:b7:c6:bb Speed:10000 Mtu:8901} {Name:b20754d576e2fb6 MacAddress:f2:0c:b3:bb:8f:25 Speed:10000 Mtu:8901} {Name:bc768d9eab33052 MacAddress:9a:59:36:8b:53:85 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f0:7a:c5:67 Speed:0 Mtu:9001} {Name:br-int MacAddress:b2:2e:b2:e7:a1:ef Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:f0:7a:c5:67 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:de:3b:e3:5d:d5:49 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c2:3d:2b:0b:cc:b1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 16:14:16.508976 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.507470 4947 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 16:14:16.508976 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.507520 4947 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 16:14:16.508976 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.507814 4947 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 16:14:16.508976 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.507840 4947 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-64.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 16:14:16.508976 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.507970 4947 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 16:14:16.508976 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.507977 4947 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 16:14:16.508976 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.507989 4947 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:14:16.508976 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.508002 4947 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 16:14:16.508976 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.508084 4947 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 16:14:16.508976 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.508246 4947 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 16:14:16.508976 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.508375 4947 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 16:14:16.508976 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.508409 4947 kubelet.go:493] "Attempting to sync node with API server" Nov 03 16:14:16.508976 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.508418 4947 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 16:14:16.508976 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.508434 4947 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 16:14:16.508976 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.508444 4947 kubelet.go:405] "Adding apiserver pod source" Nov 03 16:14:16.508976 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.508451 4947 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 16:14:16.509967 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.509831 4947 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 16:14:16.509967 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.509851 4947 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:14:16.510498 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.510476 4947 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 16:14:16.513298 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513281 4947 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 16:14:16.513298 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513296 4947 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 16:14:16.513421 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513410 4947 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 16:14:16.513617 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513606 4947 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 16:14:16.513648 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513636 4947 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 16:14:16.513769 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513759 4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 16:14:16.513800 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513777 4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 16:14:16.513800 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513783 4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 16:14:16.513800 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513789 4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 16:14:16.513800 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513795 4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 16:14:16.513911 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513802 4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 16:14:16.513911 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513812 4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 16:14:16.513911 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513818 4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 16:14:16.513911 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513824 4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 16:14:16.513911 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513834 4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 16:14:16.513911 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513848 4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 16:14:16.513911 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513857 4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 16:14:16.513911 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513884 4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 16:14:16.513911 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.513892 4947 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 16:14:16.517409 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.517392 4947 server.go:1263] "Started kubelet" Nov 03 16:14:16.517522 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.517468 4947 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 16:14:16.517592 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.517505 4947 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 16:14:16.517592 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.517585 4947 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 16:14:16.518009 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.517981 4947 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 16:14:16.518758 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.518737 4947 server.go:317] "Adding debug handlers to kubelet server" Nov 03 16:14:16.519073 ip-10-0-143-64 systemd[1]: Started Kubernetes Kubelet. Nov 03 16:14:16.520493 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.520475 4947 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 16:14:16.520493 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.520490 4947 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 16:14:16.522745 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.520573 4947 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 16:08:38 +0000 UTC" deadline="2028-06-09 20:16:27.216789982 +0000 UTC" Nov 03 16:14:16.522745 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.520608 4947 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14020h2m10.69618559s" Nov 03 16:14:16.522745 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:16.520637 4947 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-64.ec2.internal\" not found" Nov 03 16:14:16.522745 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.521229 4947 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 16:14:16.522745 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.521245 4947 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 16:14:16.522745 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.521347 4947 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 16:14:16.526728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.526107 4947 factory.go:153] Registering CRI-O factory Nov 03 16:14:16.526728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.526138 4947 factory.go:223] Registration of the crio container factory successfully Nov 03 16:14:16.526728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.526186 4947 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 16:14:16.526728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.526195 4947 factory.go:55] Registering systemd factory Nov 03 16:14:16.526728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.526203 4947 factory.go:223] Registration of the systemd container factory successfully Nov 03 16:14:16.526728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.526221 4947 factory.go:103] Registering Raw factory Nov 03 16:14:16.526728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.526237 4947 manager.go:1199] Started watching for new ooms in manager Nov 03 16:14:16.526728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.526627 4947 manager.go:319] Starting recovery of all containers Nov 03 16:14:16.530630 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530582 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ced7a93-d9f0-44de-9a76-34a84e41843b" volumeName="kubernetes.io/projected/5ced7a93-d9f0-44de-9a76-34a84e41843b-kube-api-access-przlh" seLinuxMountContext="" Nov 03 16:14:16.530630 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530628 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03cc593d-32bc-4453-8a45-2b546f862acf" volumeName="kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-kube-api-access-rl4vw" seLinuxMountContext="" Nov 03 16:14:16.530805 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530639 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03cc593d-32bc-4453-8a45-2b546f862acf" volumeName="kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-bound-sa-token" seLinuxMountContext="" Nov 03 16:14:16.530805 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530652 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1580792e-eb86-4d79-92a7-8055e6a8aaff" volumeName="kubernetes.io/secret/1580792e-eb86-4d79-92a7-8055e6a8aaff-serving-cert" seLinuxMountContext="" Nov 03 16:14:16.530805 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530661 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78e9f545-8400-48b6-968d-eb65dc94b471" volumeName="kubernetes.io/projected/78e9f545-8400-48b6-968d-eb65dc94b471-kube-api-access-9g2jb" seLinuxMountContext="" Nov 03 16:14:16.530805 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530671 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a13e4eb2-b4f2-4f66-85d3-8bedb078823c" volumeName="kubernetes.io/empty-dir/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-ca-trust-extracted" seLinuxMountContext="" Nov 03 16:14:16.530805 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530679 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be60fb17-308e-420d-b9e0-d9a05c0c752f" volumeName="kubernetes.io/secret/be60fb17-308e-420d-b9e0-d9a05c0c752f-agent-certs" seLinuxMountContext="" Nov 03 16:14:16.530805 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530687 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2ded168-1814-4922-9eee-00caf304b8af" volumeName="kubernetes.io/projected/c2ded168-1814-4922-9eee-00caf304b8af-kube-api-access-svnps" seLinuxMountContext="" Nov 03 16:14:16.530805 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530807 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03cc593d-32bc-4453-8a45-2b546f862acf" volumeName="kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-trusted-ca" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530820 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03cc593d-32bc-4453-8a45-2b546f862acf" volumeName="kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-image-registry-private-configuration" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530836 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ec02fd2-78a5-4813-b6e4-4edefa217fe1" volumeName="kubernetes.io/configmap/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-env-overrides" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530849 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28" volumeName="kubernetes.io/configmap/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-telemetry-config" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530866 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec0f2c68-a3bf-4475-ad0d-43346dbd32fd" volumeName="kubernetes.io/projected/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd-kube-api-access-5jjxv" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530889 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1566c753-13b1-4495-9268-05e3676d82bd" volumeName="kubernetes.io/configmap/1566c753-13b1-4495-9268-05e3676d82bd-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530911 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a13e4eb2-b4f2-4f66-85d3-8bedb078823c" volumeName="kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-trusted-ca" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530928 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8c316f-51ce-4c69-865f-1d481ff30630" volumeName="kubernetes.io/configmap/bf8c316f-51ce-4c69-865f-1d481ff30630-config" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530939 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ced7a93-d9f0-44de-9a76-34a84e41843b" volumeName="kubernetes.io/empty-dir/5ced7a93-d9f0-44de-9a76-34a84e41843b-tmp-dir" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530958 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7731b228-5421-4121-9228-1f19a850ef9a" volumeName="kubernetes.io/configmap/7731b228-5421-4121-9228-1f19a850ef9a-cni-binary-copy" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530970 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94fb68fc-92af-4e72-824b-7092f9875560" volumeName="kubernetes.io/projected/94fb68fc-92af-4e72-824b-7092f9875560-kube-api-access-2d2cr" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.530988 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba6c03a7-1ba9-48c0-b67b-6fb26263425e" volumeName="kubernetes.io/projected/ba6c03a7-1ba9-48c0-b67b-6fb26263425e-kube-api-access-tszlg" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531001 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8c316f-51ce-4c69-865f-1d481ff30630" volumeName="kubernetes.io/projected/bf8c316f-51ce-4c69-865f-1d481ff30630-kube-api-access-gtbz7" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531017 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1566c753-13b1-4495-9268-05e3676d82bd" volumeName="kubernetes.io/projected/1566c753-13b1-4495-9268-05e3676d82bd-kube-api-access-vwcf4" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531040 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1580792e-eb86-4d79-92a7-8055e6a8aaff" volumeName="kubernetes.io/configmap/1580792e-eb86-4d79-92a7-8055e6a8aaff-config" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531050 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7f5be2e-c57e-432f-a5bb-144709f3b492" volumeName="kubernetes.io/empty-dir/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-tuned" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531061 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1566c753-13b1-4495-9268-05e3676d82bd" volumeName="kubernetes.io/configmap/1566c753-13b1-4495-9268-05e3676d82bd-cni-binary-copy" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531070 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="216c0d66-a3ec-4748-b761-3420536bfa3a" volumeName="kubernetes.io/configmap/216c0d66-a3ec-4748-b761-3420536bfa3a-trusted-ca" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531080 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="216c0d66-a3ec-4748-b761-3420536bfa3a" volumeName="kubernetes.io/projected/216c0d66-a3ec-4748-b761-3420536bfa3a-kube-api-access-wz7ht" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531088 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66b43213-9ec2-4164-9f8d-d7621bd57eb3" volumeName="kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-stats-auth" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531104 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ec02fd2-78a5-4813-b6e4-4edefa217fe1" volumeName="kubernetes.io/configmap/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-ovnkube-script-lib" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531114 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ec02fd2-78a5-4813-b6e4-4edefa217fe1" volumeName="kubernetes.io/configmap/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-ovnkube-config" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531125 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ec02fd2-78a5-4813-b6e4-4edefa217fe1" volumeName="kubernetes.io/secret/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531136 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9c106a2-ce74-4c57-80c1-b1f75da0d43d" volumeName="kubernetes.io/secret/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-serving-cert" seLinuxMountContext="" Nov 03 16:14:16.531148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531149 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28" volumeName="kubernetes.io/projected/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-kube-api-access-phmbx" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531166 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa01a744-9121-4f6a-b280-dba215c1ab8f" volumeName="kubernetes.io/projected/fa01a744-9121-4f6a-b280-dba215c1ab8f-kube-api-access-d4vdm" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531178 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03cc593d-32bc-4453-8a45-2b546f862acf" volumeName="kubernetes.io/empty-dir/03cc593d-32bc-4453-8a45-2b546f862acf-ca-trust-extracted" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531193 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ced7a93-d9f0-44de-9a76-34a84e41843b" volumeName="kubernetes.io/configmap/5ced7a93-d9f0-44de-9a76-34a84e41843b-config-volume" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531204 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66b43213-9ec2-4164-9f8d-d7621bd57eb3" volumeName="kubernetes.io/projected/66b43213-9ec2-4164-9f8d-d7621bd57eb3-kube-api-access-75xq7" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531215 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ec02fd2-78a5-4813-b6e4-4edefa217fe1" volumeName="kubernetes.io/projected/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-kube-api-access-cfv4w" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531230 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3ce380e-3ddd-42bd-867c-9bba84f2bab4" volumeName="kubernetes.io/projected/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-kube-api-access-gjwvk" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531244 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f02c434e-ede0-4bea-8830-f3444c21a6a6" volumeName="kubernetes.io/empty-dir/f02c434e-ede0-4bea-8830-f3444c21a6a6-data-volume" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531259 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f02c434e-ede0-4bea-8830-f3444c21a6a6" volumeName="kubernetes.io/projected/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-api-access-kfkpb" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531270 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a13e4eb2-b4f2-4f66-85d3-8bedb078823c" volumeName="kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-image-registry-private-configuration" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531282 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4cbc83f2-64b1-4804-a180-43df6459ffa4" volumeName="kubernetes.io/projected/4cbc83f2-64b1-4804-a180-43df6459ffa4-kube-api-access-42fxm" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531297 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="216c0d66-a3ec-4748-b761-3420536bfa3a" volumeName="kubernetes.io/secret/216c0d66-a3ec-4748-b761-3420536bfa3a-serving-cert" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531306 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4b370319-6de8-41e2-ae5f-21267a2fba74" volumeName="kubernetes.io/projected/4b370319-6de8-41e2-ae5f-21267a2fba74-kube-api-access-pzhxd" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531331 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="812067ca-eef5-4ab1-a24e-61796ac0721b" volumeName="kubernetes.io/configmap/812067ca-eef5-4ab1-a24e-61796ac0721b-serviceca" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531343 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a13e4eb2-b4f2-4f66-85d3-8bedb078823c" volumeName="kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-certificates" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531354 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7f5be2e-c57e-432f-a5bb-144709f3b492" volumeName="kubernetes.io/projected/a7f5be2e-c57e-432f-a5bb-144709f3b492-kube-api-access-j9hhm" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531371 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1566c753-13b1-4495-9268-05e3676d82bd" volumeName="kubernetes.io/configmap/1566c753-13b1-4495-9268-05e3676d82bd-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531384 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="812067ca-eef5-4ab1-a24e-61796ac0721b" volumeName="kubernetes.io/projected/812067ca-eef5-4ab1-a24e-61796ac0721b-kube-api-access-hn2hk" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531401 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a13e4eb2-b4f2-4f66-85d3-8bedb078823c" volumeName="kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-installation-pull-secrets" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531413 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9c106a2-ce74-4c57-80c1-b1f75da0d43d" volumeName="kubernetes.io/configmap/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-trusted-ca-bundle" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531424 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66b43213-9ec2-4164-9f8d-d7621bd57eb3" volumeName="kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-default-certificate" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531441 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c96f345-2cda-4554-bef3-be416dc50225" volumeName="kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531452 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a13e4eb2-b4f2-4f66-85d3-8bedb078823c" volumeName="kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-kube-api-access-z4dbx" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531467 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9c106a2-ce74-4c57-80c1-b1f75da0d43d" volumeName="kubernetes.io/configmap/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-service-ca-bundle" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531497 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cbd4c91-69f0-47be-94be-e451d44b4992" volumeName="kubernetes.io/projected/3cbd4c91-69f0-47be-94be-e451d44b4992-kube-api-access-l4h7b" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531512 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55767581-2059-49bf-8dd0-69e31d5adb83" volumeName="kubernetes.io/projected/55767581-2059-49bf-8dd0-69e31d5adb83-kube-api-access-2z647" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531527 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7731b228-5421-4121-9228-1f19a850ef9a" volumeName="kubernetes.io/configmap/7731b228-5421-4121-9228-1f19a850ef9a-multus-daemon-config" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531538 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7731b228-5421-4121-9228-1f19a850ef9a" volumeName="kubernetes.io/projected/7731b228-5421-4121-9228-1f19a850ef9a-kube-api-access-vp88d" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531553 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9c106a2-ce74-4c57-80c1-b1f75da0d43d" volumeName="kubernetes.io/projected/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-kube-api-access-4rznq" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531566 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be60fb17-308e-420d-b9e0-d9a05c0c752f" volumeName="kubernetes.io/configmap/be60fb17-308e-420d-b9e0-d9a05c0c752f-konnectivity-ca" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531578 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec0f2c68-a3bf-4475-ad0d-43346dbd32fd" volumeName="kubernetes.io/configmap/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd-iptables-alerter-script" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531596 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03cc593d-32bc-4453-8a45-2b546f862acf" volumeName="kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-installation-pull-secrets" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531609 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9c106a2-ce74-4c57-80c1-b1f75da0d43d" volumeName="kubernetes.io/empty-dir/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-snapshots" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531626 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03cc593d-32bc-4453-8a45-2b546f862acf" volumeName="kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-registry-certificates" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531639 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1580792e-eb86-4d79-92a7-8055e6a8aaff" volumeName="kubernetes.io/projected/1580792e-eb86-4d79-92a7-8055e6a8aaff-kube-api-access-8fx4n" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531651 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="216c0d66-a3ec-4748-b761-3420536bfa3a" volumeName="kubernetes.io/configmap/216c0d66-a3ec-4748-b761-3420536bfa3a-config" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531667 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cbd4c91-69f0-47be-94be-e451d44b4992" volumeName="kubernetes.io/empty-dir/3cbd4c91-69f0-47be-94be-e451d44b4992-tmp-dir" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531679 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2ded168-1814-4922-9eee-00caf304b8af" volumeName="kubernetes.io/configmap/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-config" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531714 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="110bff76-1581-4379-8468-441a3a81dea2" volumeName="kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531729 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c812633-b15d-4a30-982d-80d276ec78c7" volumeName="kubernetes.io/configmap/2c812633-b15d-4a30-982d-80d276ec78c7-nginx-conf" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531768 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f85dfa8-9158-437d-b151-d8eb8fe65b7b" volumeName="kubernetes.io/projected/8f85dfa8-9158-437d-b151-d8eb8fe65b7b-kube-api-access-xdzb4" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531787 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f85dfa8-9158-437d-b151-d8eb8fe65b7b" volumeName="kubernetes.io/secret/8f85dfa8-9158-437d-b151-d8eb8fe65b7b-cert" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531801 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7f5be2e-c57e-432f-a5bb-144709f3b492" volumeName="kubernetes.io/empty-dir/a7f5be2e-c57e-432f-a5bb-144709f3b492-tmp" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531819 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bf8c316f-51ce-4c69-865f-1d481ff30630" volumeName="kubernetes.io/secret/bf8c316f-51ce-4c69-865f-1d481ff30630-serving-cert" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531831 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a13e4eb2-b4f2-4f66-85d3-8bedb078823c" volumeName="kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-bound-sa-token" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531844 4947 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9c106a2-ce74-4c57-80c1-b1f75da0d43d" volumeName="kubernetes.io/empty-dir/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-tmp" seLinuxMountContext="" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531860 4947 reconstruct.go:97] "Volume reconstruction finished" Nov 03 16:14:16.532200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.531868 4947 reconciler.go:29] "Reconciler: start to sync state" Nov 03 16:14:16.538857 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.538804 4947 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 16:14:16.544382 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.544355 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/1.log" Nov 03 16:14:16.544607 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.544590 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/1.log" Nov 03 16:14:16.557029 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.556984 4947 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 16:14:16.557029 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.557028 4947 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 16:14:16.557255 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.557052 4947 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 16:14:16.557255 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:16.557096 4947 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 16:14:16.579084 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.579000 4947 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 16:14:16.580265 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.580238 4947 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:14:16.580408 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.580250 4947 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:14:16.585418 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.585399 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/1.log" Nov 03 16:14:16.585527 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.585423 4947 generic.go:356] "Generic (PLEG): container finished" podID="216c0d66-a3ec-4748-b761-3420536bfa3a" containerID="02633c14002a1b44b88bd7bf4802aa104ef3c6c904de369370b2ccfa72bbf740" exitCode=255 Nov 03 16:14:16.588899 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.588871 4947 generic.go:356] "Generic (PLEG): container finished" podID="10d52772a6531c8bd8223292030e42c2" containerID="e2963ede6517632a7b5156383b0cd5e7accc98704b47c2ce6afbd4da72c153d0" exitCode=0 Nov 03 16:14:16.594305 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.594279 4947 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:14:16.594439 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.594400 4947 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 16:14:16.608942 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.608914 4947 generic.go:356] "Generic (PLEG): container finished" podID="1566c753-13b1-4495-9268-05e3676d82bd" containerID="a23d50a3132cdeee005ca133ad5efbc08d4c4f76a7bd9a750c694db889baf0e3" exitCode=0 Nov 03 16:14:16.608942 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.608939 4947 generic.go:356] "Generic (PLEG): container finished" podID="1566c753-13b1-4495-9268-05e3676d82bd" containerID="15fb6ad3281402c8396ff124f31ff074812150306809ef8ef6ce26acf4e803f7" exitCode=0 Nov 03 16:14:16.608942 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.608946 4947 generic.go:356] "Generic (PLEG): container finished" podID="1566c753-13b1-4495-9268-05e3676d82bd" containerID="0c278f9154d27a09660bd01f4bc4e16bfc60ce47f219597d7b3052f5ba7c6fe7" exitCode=0 Nov 03 16:14:16.608942 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.608952 4947 generic.go:356] "Generic (PLEG): container finished" podID="1566c753-13b1-4495-9268-05e3676d82bd" containerID="40b1d60e5d8628c9955ddc19ac12381b06dd5d4048d3ce8fa90c4c37a2d8d78a" exitCode=0 Nov 03 16:14:16.609161 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.608959 4947 generic.go:356] "Generic (PLEG): container finished" podID="1566c753-13b1-4495-9268-05e3676d82bd" containerID="9b15235b362680b2f9963b27065f017a46f3543bafe13086e55060d179de848c" exitCode=0 Nov 03 16:14:16.609161 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.608964 4947 generic.go:356] "Generic (PLEG): container finished" podID="1566c753-13b1-4495-9268-05e3676d82bd" containerID="9d6f3251807b378afdba565cb46127634bbd4ff94fbdae491344df215a477101" exitCode=0 Nov 03 16:14:16.622042 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.622017 4947 manager.go:324] Recovery completed Nov 03 16:14:16.643986 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.643960 4947 cpu_manager.go:226] "Starting" policy="none" Nov 03 16:14:16.643986 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.643981 4947 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 16:14:16.644171 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.644002 4947 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 16:14:16.644171 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.644140 4947 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 16:14:16.644171 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.644151 4947 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 16:14:16.644272 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.644174 4947 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 16:14:16.644272 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.644180 4947 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 16:14:16.644272 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.644185 4947 policy_none.go:50] "Start" Nov 03 16:14:16.644272 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.644195 4947 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 16:14:16.644272 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.644204 4947 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 16:14:16.644407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.644328 4947 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 16:14:16.644407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.644335 4947 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 16:14:16.644407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.644341 4947 policy_none.go:44] "Start" Nov 03 16:14:16.648685 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.648642 4947 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 16:14:16.652495 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.652476 4947 manager.go:341] "Starting Device Plugin manager" Nov 03 16:14:16.653122 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:16.652507 4947 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 16:14:16.653199 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.653128 4947 server.go:87] "Starting device plugin registration server" Nov 03 16:14:16.653404 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.653392 4947 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 16:14:16.653435 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.653408 4947 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 16:14:16.653535 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.653504 4947 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 16:14:16.653655 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.653618 4947 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 16:14:16.653719 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.653666 4947 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 16:14:16.653719 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.653674 4947 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 16:14:16.653826 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.653741 4947 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T16:14:16.65365501Z","UUID":"38f19108-aa55-4bfc-83fd-432ff0cd3b5c","Handler":null,"Name":"","Endpoint":""} Nov 03 16:14:16.655018 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.655000 4947 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 16:14:16.655146 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.655024 4947 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 16:14:16.655598 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:16.655339 4947 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 16:14:16.657505 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.657465 4947 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal"] Nov 03 16:14:16.658876 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.658852 4947 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:14:16.659015 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.658970 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal" event={"ID":"9d137c53e9ec4edb6f2bb3a754f3bb53","Type":"ContainerStarted","Data":"fbf3dfb75efd2778a51fb0f04629ba8eec1f077924de089eec69bed67d1c2cd6"} Nov 03 16:14:16.659079 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.659014 4947 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal" Nov 03 16:14:16.659079 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.659039 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal" event={"ID":"9d137c53e9ec4edb6f2bb3a754f3bb53","Type":"ContainerStarted","Data":"cd11594e79cefb0bb847b42f8639cfaf6757665400c707c490511b666e78a220"} Nov 03 16:14:16.659079 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.659070 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" event={"ID":"10d52772a6531c8bd8223292030e42c2","Type":"ContainerStarted","Data":"2669f9756dffea804eb5f9a02d0538ed5c338c00266f588522d1668afca26f1e"} Nov 03 16:14:16.659217 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.659094 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" event={"ID":"10d52772a6531c8bd8223292030e42c2","Type":"ContainerDied","Data":"e2963ede6517632a7b5156383b0cd5e7accc98704b47c2ce6afbd4da72c153d0"} Nov 03 16:14:16.659217 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.659123 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" event={"ID":"10d52772a6531c8bd8223292030e42c2","Type":"ContainerStarted","Data":"2c6d7e35341f617447534ea331414166e75c5c6220a636690c5e2db938cc4ad7"} Nov 03 16:14:16.692821 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.692796 4947 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:14:16.692981 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:16.692867 4947 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-64.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal" Nov 03 16:14:16.696551 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.696537 4947 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:14:16.696615 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:16.696575 4947 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:14:16.753958 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.753923 4947 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 16:14:16.755311 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.755292 4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasSufficientMemory" Nov 03 16:14:16.755375 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.755325 4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasNoDiskPressure" Nov 03 16:14:16.755375 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.755335 4947 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-64.ec2.internal" event="NodeHasSufficientPID" Nov 03 16:14:16.755430 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.755376 4947 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-64.ec2.internal" Nov 03 16:14:16.780182 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.780154 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10d52772a6531c8bd8223292030e42c2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal\" (UID: \"10d52772a6531c8bd8223292030e42c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:14:16.780292 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.780200 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9d137c53e9ec4edb6f2bb3a754f3bb53-config\") pod \"kube-apiserver-proxy-ip-10-0-143-64.ec2.internal\" (UID: \"9d137c53e9ec4edb6f2bb3a754f3bb53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal" Nov 03 16:14:16.780292 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.780226 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10d52772a6531c8bd8223292030e42c2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal\" (UID: \"10d52772a6531c8bd8223292030e42c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:14:16.793678 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.793657 4947 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-64.ec2.internal" Nov 03 16:14:16.793752 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.793742 4947 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-64.ec2.internal" Nov 03 16:14:16.881171 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.881098 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10d52772a6531c8bd8223292030e42c2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal\" (UID: \"10d52772a6531c8bd8223292030e42c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:14:16.881171 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.881130 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10d52772a6531c8bd8223292030e42c2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal\" (UID: \"10d52772a6531c8bd8223292030e42c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:14:16.881171 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.881154 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9d137c53e9ec4edb6f2bb3a754f3bb53-config\") pod \"kube-apiserver-proxy-ip-10-0-143-64.ec2.internal\" (UID: \"9d137c53e9ec4edb6f2bb3a754f3bb53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal" Nov 03 16:14:16.881368 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.881204 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9d137c53e9ec4edb6f2bb3a754f3bb53-config\") pod \"kube-apiserver-proxy-ip-10-0-143-64.ec2.internal\" (UID: \"9d137c53e9ec4edb6f2bb3a754f3bb53\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal" Nov 03 16:14:16.881368 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.881220 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10d52772a6531c8bd8223292030e42c2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal\" (UID: \"10d52772a6531c8bd8223292030e42c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:14:16.881368 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:16.881221 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10d52772a6531c8bd8223292030e42c2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal\" (UID: \"10d52772a6531c8bd8223292030e42c2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:14:17.508797 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.508755 4947 apiserver.go:52] "Watching apiserver" Nov 03 16:14:17.553565 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.553529 4947 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 16:14:17.554371 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.554347 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal","openshift-cluster-node-tuning-operator/tuned-2kl7n","openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft","openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt","openshift-kube-storage-version-migrator/migrator-66df5d496b-g8vzv","openshift-insights/insights-operator-7c4986d449-964q9","openshift-multus/multus-additional-cni-plugins-r6mtg","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4","openshift-image-registry/image-registry-68c8fc97c8-vxzz9","openshift-console-operator/console-operator-7cd56b8dbc-tnh8w","openshift-network-operator/iptables-alerter-g5w4j","openshift-insights/insights-runtime-extractor-rj5rd","openshift-network-diagnostics/network-check-source-6f78d59f99-cm2k2","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-88dnt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal","openshift-network-console/networking-console-plugin-86c478bb88-r755z","openshift-network-diagnostics/network-check-target-5fq8q","kube-system/konnectivity-agent-zmzv6","kube-system/global-pull-secret-syncer-p7vvq","openshift-image-registry/image-registry-7d945bd975-wtl6x","openshift-dns/node-resolver-zfshj","openshift-image-registry/node-ca-bflh2","openshift-ingress-canary/ingress-canary-jdthd","openshift-multus/multus-kmfdd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g","kube-system/kas-connection-checker-685895c7c9-4j5k5","openshift-ovn-kubernetes/ovnkube-node-gvrm4","kube-system/kas-connection-checker-685895c7c9-hxr8b","kube-system/kas-connection-checker-685895c7c9-j8t6k","openshift-ingress/router-default-5dc56589f5-wsj6w","openshift-multus/network-metrics-daemon-n7hw7","openshift-dns/dns-default-xfk72"] Nov 03 16:14:17.557993 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.557957 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:17.559365 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.559344 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:17.559691 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.559671 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:17.560378 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.560362 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:17.561401 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.561384 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:17.562484 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.562468 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:14:17.566272 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.566252 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:17.568918 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.568897 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:17.569558 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.569538 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:17.574080 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.574062 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.574358 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.574346 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.577556 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.577540 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.577891 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.577873 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.579278 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.579258 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 16:14:17.595449 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.595417 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 16:14:17.595449 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.595417 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 16:14:17.595653 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.595423 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 16:14:17.602976 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.602943 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 16:14:17.603126 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.602999 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 16:14:17.603471 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.603447 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 16:14:17.603578 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.603492 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 16:14:17.603916 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.603848 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 16:14:17.603916 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.603874 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.604056 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.603924 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.604056 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.603973 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 16:14:17.604056 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.603999 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.604056 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.603998 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 16:14:17.604056 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604046 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6q76v\"" Nov 03 16:14:17.604472 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604073 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.604472 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604118 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 16:14:17.604472 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604424 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 16:14:17.604472 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604424 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 16:14:17.604472 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604450 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.604472 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604458 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 16:14:17.604802 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604519 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6xtgh\"" Nov 03 16:14:17.604802 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604552 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 16:14:17.604802 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604563 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.604802 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604554 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.604802 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604554 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 16:14:17.604802 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604586 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.604802 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604605 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.604802 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604635 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604871 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604891 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604904 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604891 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5xwqx\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604920 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604938 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604892 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604941 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-d6vzh\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604943 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604970 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n246q\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604981 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604982 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hxhwt\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604993 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.604973 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605002 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605097 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605119 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605145 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q9tss\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605146 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605174 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605184 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605185 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605178 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.605316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605222 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.606230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605377 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-67www\"" Nov 03 16:14:17.606230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605423 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tn5d9\"" Nov 03 16:14:17.606230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605437 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-7jhdd\"" Nov 03 16:14:17.606230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605467 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nnkq7\"" Nov 03 16:14:17.606230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605466 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s5c6x\"" Nov 03 16:14:17.606230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605469 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.606230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605480 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.606230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605532 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.606230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605559 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2k4df\"" Nov 03 16:14:17.606230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605569 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 16:14:17.606230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605664 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tf624\"" Nov 03 16:14:17.606230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605722 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 16:14:17.606230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.605775 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 16:14:17.608807 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.608779 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 16:14:17.609355 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.609337 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.609773 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.609752 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.609851 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.609758 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.609851 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.609814 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qj6r5\"" Nov 03 16:14:17.609971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.609864 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.609971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.609886 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5snks\"" Nov 03 16:14:17.609971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.609887 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 16:14:17.609971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.609915 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 16:14:17.609971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.609933 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6cp7r\"" Nov 03 16:14:17.609971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.609956 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.610192 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.609973 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 16:14:17.610192 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.610037 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.613860 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.613841 4947 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:14:17.613981 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.613968 4947 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal" Nov 03 16:14:17.617770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.617737 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.617879 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.617789 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 16:14:17.617879 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.617800 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 16:14:17.617879 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.617804 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.617879 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.617827 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.617879 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.617836 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8mlmt\"" Nov 03 16:14:17.617879 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.617794 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.618070 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.617896 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.618070 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.617949 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 16:14:17.618317 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.618299 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 16:14:17.618416 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.618393 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.618416 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.618407 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.618416 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.618313 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 16:14:17.618416 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.618335 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 16:14:17.618604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.618306 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 16:14:17.618604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.618336 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 16:14:17.619004 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.618987 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 16:14:17.623554 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.623538 4947 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 16:14:17.638900 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.638876 4947 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:14:17.639011 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.638929 4947 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-64.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-64.ec2.internal" Nov 03 16:14:17.639196 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.639178 4947 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 16:14:17.639241 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.639222 4947 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-64.ec2.internal" Nov 03 16:14:17.685528 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.685496 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-var-lib-cni-bin\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.685629 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.685536 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-installation-pull-secrets\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:17.685629 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.685555 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd-iptables-alerter-script\") pod \"iptables-alerter-g5w4j\" (UID: \"ec0f2c68-a3bf-4475-ad0d-43346dbd32fd\") " pod="openshift-network-operator/iptables-alerter-g5w4j" Nov 03 16:14:17.685629 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.685618 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ced7a93-d9f0-44de-9a76-34a84e41843b-tmp-dir\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:17.685772 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.685651 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-przlh\" (UniqueName: \"kubernetes.io/projected/5ced7a93-d9f0-44de-9a76-34a84e41843b-kube-api-access-przlh\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:17.685822 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.685778 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:14:17.685822 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.685784 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ced7a93-d9f0-44de-9a76-34a84e41843b-tmp-dir\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:17.685885 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.685846 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:17.685885 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.685859 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-installation-pull-secrets\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:17.685885 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.685870 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-registry-certificates\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:17.686009 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.685891 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-device-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.686009 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.685922 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-cni-netd\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.686009 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.685943 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-trusted-ca\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:17.686009 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.685954 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd-iptables-alerter-script\") pod \"iptables-alerter-g5w4j\" (UID: \"ec0f2c68-a3bf-4475-ad0d-43346dbd32fd\") " pod="openshift-network-operator/iptables-alerter-g5w4j" Nov 03 16:14:17.686009 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.685995 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-modprobe-d\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.686247 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.686016 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f02c434e-ede0-4bea-8830-f3444c21a6a6-crio-socket\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:17.686247 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.686034 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:14:17.686247 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.686131 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-registry-certificates\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:17.686247 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.686140 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cbd4c91-69f0-47be-94be-e451d44b4992-tmp-dir\") pod \"node-resolver-zfshj\" (UID: \"3cbd4c91-69f0-47be-94be-e451d44b4992\") " pod="openshift-dns/node-resolver-zfshj" Nov 03 16:14:17.686247 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.686186 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdzb4\" (UniqueName: \"kubernetes.io/projected/8f85dfa8-9158-437d-b151-d8eb8fe65b7b-kube-api-access-xdzb4\") pod \"ingress-canary-jdthd\" (UID: \"8f85dfa8-9158-437d-b151-d8eb8fe65b7b\") " pod="openshift-ingress-canary/ingress-canary-jdthd" Nov 03 16:14:17.686247 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.686210 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cbd4c91-69f0-47be-94be-e451d44b4992-tmp-dir\") pod \"node-resolver-zfshj\" (UID: \"3cbd4c91-69f0-47be-94be-e451d44b4992\") " pod="openshift-dns/node-resolver-zfshj" Nov 03 16:14:17.686247 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.686219 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vdm\" (UniqueName: \"kubernetes.io/projected/fa01a744-9121-4f6a-b280-dba215c1ab8f-kube-api-access-d4vdm\") pod \"volume-data-source-validator-58c9f5f46d-88dnt\" (UID: \"fa01a744-9121-4f6a-b280-dba215c1ab8f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-88dnt" Nov 03 16:14:17.686662 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.686599 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-snapshots\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:17.686662 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.686645 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-trusted-ca\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:17.686827 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.686254 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-snapshots\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:17.686827 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.686664 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c96f345-2cda-4554-bef3-be416dc50225-original-pull-secret\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:14:17.686827 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.686761 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-systemd-units\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.686827 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.686802 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfkpb\" (UniqueName: \"kubernetes.io/projected/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-api-access-kfkpb\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:17.687002 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.686864 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:17.687002 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.686940 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-ovn-node-metrics-cert\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.687002 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.686980 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-var-lib-cni-multus\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.687132 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687016 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-run-multus-certs\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.687132 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687053 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp88d\" (UniqueName: \"kubernetes.io/projected/7731b228-5421-4121-9228-1f19a850ef9a-kube-api-access-vp88d\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.687132 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687082 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7f5be2e-c57e-432f-a5bb-144709f3b492-tmp\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.687132 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687117 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5ced7a93-d9f0-44de-9a76-34a84e41843b-config-volume\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:17.687301 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687142 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:17.687301 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687150 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/be60fb17-308e-420d-b9e0-d9a05c0c752f-konnectivity-ca\") pod \"konnectivity-agent-zmzv6\" (UID: \"be60fb17-308e-420d-b9e0-d9a05c0c752f\") " pod="kube-system/konnectivity-agent-zmzv6" Nov 03 16:14:17.687301 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687182 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c96f345-2cda-4554-bef3-be416dc50225-kubelet-config\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:14:17.687581 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687552 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf8c316f-51ce-4c69-865f-1d481ff30630-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4\" (UID: \"bf8c316f-51ce-4c69-865f-1d481ff30630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" Nov 03 16:14:17.687724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687602 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-ovn-node-metrics-cert\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.687724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687606 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/216c0d66-a3ec-4748-b761-3420536bfa3a-serving-cert\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:17.687724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687633 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a7f5be2e-c57e-432f-a5bb-144709f3b492-tmp\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.687724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687691 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1566c753-13b1-4495-9268-05e3676d82bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:14:17.687937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687754 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1566c753-13b1-4495-9268-05e3676d82bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:14:17.687937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687791 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1580792e-eb86-4d79-92a7-8055e6a8aaff-config\") pod \"service-ca-operator-544d749c8c-qqcft\" (UID: \"1580792e-eb86-4d79-92a7-8055e6a8aaff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" Nov 03 16:14:17.687937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687822 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5ced7a93-d9f0-44de-9a76-34a84e41843b-config-volume\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:17.687937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687887 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/216c0d66-a3ec-4748-b761-3420536bfa3a-serving-cert\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:17.687937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687888 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfv4w\" (UniqueName: \"kubernetes.io/projected/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-kube-api-access-cfv4w\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.688170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687948 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4h7b\" (UniqueName: \"kubernetes.io/projected/3cbd4c91-69f0-47be-94be-e451d44b4992-kube-api-access-l4h7b\") pod \"node-resolver-zfshj\" (UID: \"3cbd4c91-69f0-47be-94be-e451d44b4992\") " pod="openshift-dns/node-resolver-zfshj" Nov 03 16:14:17.688170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687987 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:17.688170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.687991 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/be60fb17-308e-420d-b9e0-d9a05c0c752f-konnectivity-ca\") pod \"konnectivity-agent-zmzv6\" (UID: \"be60fb17-308e-420d-b9e0-d9a05c0c752f\") " pod="kube-system/konnectivity-agent-zmzv6" Nov 03 16:14:17.688170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688021 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tszlg\" (UniqueName: \"kubernetes.io/projected/ba6c03a7-1ba9-48c0-b67b-6fb26263425e-kube-api-access-tszlg\") pod \"network-check-source-6f78d59f99-cm2k2\" (UID: \"ba6c03a7-1ba9-48c0-b67b-6fb26263425e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm2k2" Nov 03 16:14:17.688170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688092 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:17.688170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688128 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-run-openvswitch\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.688170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688152 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.688506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688182 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd-host-slash\") pod \"iptables-alerter-g5w4j\" (UID: \"ec0f2c68-a3bf-4475-ad0d-43346dbd32fd\") " pod="openshift-network-operator/iptables-alerter-g5w4j" Nov 03 16:14:17.688506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688216 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-var-lib-kubelet\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.688506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688256 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z647\" (UniqueName: \"kubernetes.io/projected/55767581-2059-49bf-8dd0-69e31d5adb83-kube-api-access-2z647\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:14:17.688506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688312 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1566c753-13b1-4495-9268-05e3676d82bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:14:17.688506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688337 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-multus-conf-dir\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.688506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688354 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bf8c316f-51ce-4c69-865f-1d481ff30630-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4\" (UID: \"bf8c316f-51ce-4c69-865f-1d481ff30630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" Nov 03 16:14:17.688506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688373 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-tuned\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.688506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688459 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-tuned\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.688897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688610 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1580792e-eb86-4d79-92a7-8055e6a8aaff-config\") pod \"service-ca-operator-544d749c8c-qqcft\" (UID: \"1580792e-eb86-4d79-92a7-8055e6a8aaff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" Nov 03 16:14:17.688897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688615 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1566c753-13b1-4495-9268-05e3676d82bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:14:17.688897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688650 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-tmp\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:17.688897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688729 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-service-ca-bundle\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:17.688897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688789 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-slash\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.688897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688853 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-tmp\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:17.689167 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688950 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-service-ca-bundle\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:17.689167 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.688966 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-certificates\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:17.689167 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.689016 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2c812633-b15d-4a30-982d-80d276ec78c7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-r755z\" (UID: \"2c812633-b15d-4a30-982d-80d276ec78c7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:17.689167 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.689103 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1566c753-13b1-4495-9268-05e3676d82bd-system-cni-dir\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:14:17.689355 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.689167 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-certificates\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:17.689355 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.689198 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn2hk\" (UniqueName: \"kubernetes.io/projected/812067ca-eef5-4ab1-a24e-61796ac0721b-kube-api-access-hn2hk\") pod \"node-ca-bflh2\" (UID: \"812067ca-eef5-4ab1-a24e-61796ac0721b\") " pod="openshift-image-registry/node-ca-bflh2" Nov 03 16:14:17.689355 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.689274 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-kubelet\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.689355 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.689302 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2c812633-b15d-4a30-982d-80d276ec78c7-nginx-conf\") pod \"networking-console-plugin-86c478bb88-r755z\" (UID: \"2c812633-b15d-4a30-982d-80d276ec78c7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:17.689355 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.689324 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-lib-modules\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.689588 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.689361 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-trusted-ca\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:17.689588 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.689404 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42fxm\" (UniqueName: \"kubernetes.io/projected/4cbc83f2-64b1-4804-a180-43df6459ffa4-kube-api-access-42fxm\") pod \"kas-connection-checker-685895c7c9-4j5k5\" (UID: \"4cbc83f2-64b1-4804-a180-43df6459ffa4\") " pod="kube-system/kas-connection-checker-685895c7c9-4j5k5" Nov 03 16:14:17.689588 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.689455 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtbz7\" (UniqueName: \"kubernetes.io/projected/bf8c316f-51ce-4c69-865f-1d481ff30630-kube-api-access-gtbz7\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4\" (UID: \"bf8c316f-51ce-4c69-865f-1d481ff30630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" Nov 03 16:14:17.689588 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.689501 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-bound-sa-token\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:17.689824 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.689630 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-systemd\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.689824 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.689673 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c96f345-2cda-4554-bef3-be416dc50225-dbus\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:14:17.689824 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.689744 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-run-netns\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.689824 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.689774 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-etc-openvswitch\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.689824 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.689816 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/216c0d66-a3ec-4748-b761-3420536bfa3a-trusted-ca\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:17.690130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.689887 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.690936 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.690348 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-trusted-ca\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:17.690936 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.690452 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9hhm\" (UniqueName: \"kubernetes.io/projected/a7f5be2e-c57e-432f-a5bb-144709f3b492-kube-api-access-j9hhm\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.690936 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.690507 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwcf4\" (UniqueName: \"kubernetes.io/projected/1566c753-13b1-4495-9268-05e3676d82bd-kube-api-access-vwcf4\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:14:17.690936 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.690689 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/216c0d66-a3ec-4748-b761-3420536bfa3a-trusted-ca\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:17.690936 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.690781 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d2cr\" (UniqueName: \"kubernetes.io/projected/94fb68fc-92af-4e72-824b-7092f9875560-kube-api-access-2d2cr\") pod \"migrator-66df5d496b-g8vzv\" (UID: \"94fb68fc-92af-4e72-824b-7092f9875560\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g8vzv" Nov 03 16:14:17.690936 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.690824 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:17.690936 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.690860 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1566c753-13b1-4495-9268-05e3676d82bd-cnibin\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:14:17.690936 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.690889 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf8c316f-51ce-4c69-865f-1d481ff30630-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4\" (UID: \"bf8c316f-51ce-4c69-865f-1d481ff30630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" Nov 03 16:14:17.690936 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.690939 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-system-cni-dir\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.691517 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.691100 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf8c316f-51ce-4c69-865f-1d481ff30630-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4\" (UID: \"bf8c316f-51ce-4c69-865f-1d481ff30630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" Nov 03 16:14:17.691517 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.691150 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7731b228-5421-4121-9228-1f19a850ef9a-cni-binary-copy\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.691517 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.691229 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phmbx\" (UniqueName: \"kubernetes.io/projected/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-kube-api-access-phmbx\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:17.691517 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.691284 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz7ht\" (UniqueName: \"kubernetes.io/projected/216c0d66-a3ec-4748-b761-3420536bfa3a-kube-api-access-wz7ht\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:17.691517 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.691408 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-multus-socket-dir-parent\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.691517 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.691438 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7731b228-5421-4121-9228-1f19a850ef9a-multus-daemon-config\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.691517 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.691481 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7731b228-5421-4121-9228-1f19a850ef9a-cni-binary-copy\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.691517 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.691514 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:17.691938 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.691550 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-etc-kubernetes\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.691938 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.691606 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-sys\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.691938 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.691637 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-run-systemd\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.691938 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.691715 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-ca-trust-extracted\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:17.691938 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.691746 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rznq\" (UniqueName: \"kubernetes.io/projected/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-kube-api-access-4rznq\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:17.691938 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.691853 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-ca-trust-extracted\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:17.691938 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.691929 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7731b228-5421-4121-9228-1f19a850ef9a-multus-daemon-config\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.691938 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.691935 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:17.692263 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.691971 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f02c434e-ede0-4bea-8830-f3444c21a6a6-data-volume\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:17.692263 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.692091 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:17.692263 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.692168 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f02c434e-ede0-4bea-8830-f3444c21a6a6-data-volume\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:17.692263 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.692253 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-etc-selinux\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.692407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.692302 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-hostroot\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.692407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.692350 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-stats-auth\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:17.692811 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.692790 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-kubernetes\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.692853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.692839 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-serving-cert\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:17.692890 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.692872 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:17.692926 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.692901 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1580792e-eb86-4d79-92a7-8055e6a8aaff-serving-cert\") pod \"service-ca-operator-544d749c8c-qqcft\" (UID: \"1580792e-eb86-4d79-92a7-8055e6a8aaff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" Nov 03 16:14:17.693004 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.692927 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03cc593d-32bc-4453-8a45-2b546f862acf-ca-trust-extracted\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:17.693004 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.692954 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-socket-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.693004 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.692979 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.693004 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.692989 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-stats-auth\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:17.693165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693004 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/216c0d66-a3ec-4748-b761-3420536bfa3a-config\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:17.693165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693063 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-cnibin\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.693165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693077 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03cc593d-32bc-4453-8a45-2b546f862acf-ca-trust-extracted\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:17.693165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693075 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1580792e-eb86-4d79-92a7-8055e6a8aaff-serving-cert\") pod \"service-ca-operator-544d749c8c-qqcft\" (UID: \"1580792e-eb86-4d79-92a7-8055e6a8aaff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" Nov 03 16:14:17.693165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693069 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-serving-cert\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:17.693165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693127 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1566c753-13b1-4495-9268-05e3676d82bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:14:17.693165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693160 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/be60fb17-308e-420d-b9e0-d9a05c0c752f-agent-certs\") pod \"konnectivity-agent-zmzv6\" (UID: \"be60fb17-308e-420d-b9e0-d9a05c0c752f\") " pod="kube-system/konnectivity-agent-zmzv6" Nov 03 16:14:17.693446 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693188 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-installation-pull-secrets\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:17.693446 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693212 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/216c0d66-a3ec-4748-b761-3420536bfa3a-config\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:17.693446 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693214 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl4vw\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-kube-api-access-rl4vw\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:17.693446 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693249 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1566c753-13b1-4495-9268-05e3676d82bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:14:17.693446 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693259 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-var-lib-openvswitch\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.693446 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693290 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xq7\" (UniqueName: \"kubernetes.io/projected/66b43213-9ec2-4164-9f8d-d7621bd57eb3-kube-api-access-75xq7\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:17.693446 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693317 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:17.693446 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693346 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9slx\" (UniqueName: \"kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx\") pod \"network-check-target-5fq8q\" (UID: \"110bff76-1581-4379-8468-441a3a81dea2\") " pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:14:17.693446 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693368 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:17.693446 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693394 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzhxd\" (UniqueName: \"kubernetes.io/projected/4b370319-6de8-41e2-ae5f-21267a2fba74-kube-api-access-pzhxd\") pod \"kas-connection-checker-685895c7c9-hxr8b\" (UID: \"4b370319-6de8-41e2-ae5f-21267a2fba74\") " pod="kube-system/kas-connection-checker-685895c7c9-hxr8b" Nov 03 16:14:17.693446 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693422 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-installation-pull-secrets\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:17.693446 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693429 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g2jb\" (UniqueName: \"kubernetes.io/projected/78e9f545-8400-48b6-968d-eb65dc94b471-kube-api-access-9g2jb\") pod \"kas-connection-checker-685895c7c9-j8t6k\" (UID: \"78e9f545-8400-48b6-968d-eb65dc94b471\") " pod="kube-system/kas-connection-checker-685895c7c9-j8t6k" Nov 03 16:14:17.693971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693455 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-multus-cni-dir\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.693971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693478 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cbd4c91-69f0-47be-94be-e451d44b4992-hosts-file\") pod \"node-resolver-zfshj\" (UID: \"3cbd4c91-69f0-47be-94be-e451d44b4992\") " pod="openshift-dns/node-resolver-zfshj" Nov 03 16:14:17.693971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693523 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:17.693971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693559 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/be60fb17-308e-420d-b9e0-d9a05c0c752f-agent-certs\") pod \"konnectivity-agent-zmzv6\" (UID: \"be60fb17-308e-420d-b9e0-d9a05c0c752f\") " pod="kube-system/konnectivity-agent-zmzv6" Nov 03 16:14:17.693971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693580 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svnps\" (UniqueName: \"kubernetes.io/projected/c2ded168-1814-4922-9eee-00caf304b8af-kube-api-access-svnps\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:17.693971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693619 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-sysctl-d\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.693971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693658 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:17.693971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693670 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1566c753-13b1-4495-9268-05e3676d82bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:14:17.693971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693723 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-run-ovn\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.693971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693762 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-run-k8s-cni-cncf-io\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.693971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693791 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-var-lib-kubelet\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.693971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693816 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/812067ca-eef5-4ab1-a24e-61796ac0721b-serviceca\") pod \"node-ca-bflh2\" (UID: \"812067ca-eef5-4ab1-a24e-61796ac0721b\") " pod="openshift-image-registry/node-ca-bflh2" Nov 03 16:14:17.693971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693835 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-registration-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.693971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693871 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjwvk\" (UniqueName: \"kubernetes.io/projected/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-kube-api-access-gjwvk\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.693971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693888 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-log-socket\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.693971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693905 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4dbx\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-kube-api-access-z4dbx\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:17.693971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.693931 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694033 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/812067ca-eef5-4ab1-a24e-61796ac0721b-serviceca\") pod \"node-ca-bflh2\" (UID: \"812067ca-eef5-4ab1-a24e-61796ac0721b\") " pod="openshift-image-registry/node-ca-bflh2" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694041 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-host\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694065 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-node-log\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694099 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-env-overrides\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694138 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f85dfa8-9158-437d-b151-d8eb8fe65b7b-cert\") pod \"ingress-canary-jdthd\" (UID: \"8f85dfa8-9158-437d-b151-d8eb8fe65b7b\") " pod="openshift-ingress-canary/ingress-canary-jdthd" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694179 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fx4n\" (UniqueName: \"kubernetes.io/projected/1580792e-eb86-4d79-92a7-8055e6a8aaff-kube-api-access-8fx4n\") pod \"service-ca-operator-544d749c8c-qqcft\" (UID: \"1580792e-eb86-4d79-92a7-8055e6a8aaff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694208 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-sys-fs\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694233 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-cni-bin\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694262 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-env-overrides\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694281 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-ovnkube-script-lib\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694363 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f85dfa8-9158-437d-b151-d8eb8fe65b7b-cert\") pod \"ingress-canary-jdthd\" (UID: \"8f85dfa8-9158-437d-b151-d8eb8fe65b7b\") " pod="openshift-ingress-canary/ingress-canary-jdthd" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694401 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-run-netns\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694422 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-image-registry-private-configuration\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694439 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-default-certificate\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694455 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-sysctl-conf\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694473 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-image-registry-private-configuration\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694483 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-ovnkube-script-lib\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694501 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-ovnkube-config\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694537 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jjxv\" (UniqueName: \"kubernetes.io/projected/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd-kube-api-access-5jjxv\") pod \"iptables-alerter-g5w4j\" (UID: \"ec0f2c68-a3bf-4475-ad0d-43346dbd32fd\") " pod="openshift-network-operator/iptables-alerter-g5w4j" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694556 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r755z\" (UID: \"2c812633-b15d-4a30-982d-80d276ec78c7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694581 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-bound-sa-token\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:17.694621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694584 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-image-registry-private-configuration\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:17.695344 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694637 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-sysconfig\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.695344 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694674 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-run\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.695344 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694714 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/812067ca-eef5-4ab1-a24e-61796ac0721b-host\") pod \"node-ca-bflh2\" (UID: \"812067ca-eef5-4ab1-a24e-61796ac0721b\") " pod="openshift-image-registry/node-ca-bflh2" Nov 03 16:14:17.695344 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694689 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-image-registry-private-configuration\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:17.695344 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694729 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-default-certificate\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:17.695344 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.694784 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-ovnkube-config\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.718640 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.718598 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl4vw\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-kube-api-access-rl4vw\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:17.726460 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.726430 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-przlh\" (UniqueName: \"kubernetes.io/projected/5ced7a93-d9f0-44de-9a76-34a84e41843b-kube-api-access-przlh\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:17.726752 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.726732 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rznq\" (UniqueName: \"kubernetes.io/projected/b9c106a2-ce74-4c57-80c1-b1f75da0d43d-kube-api-access-4rznq\") pod \"insights-operator-7c4986d449-964q9\" (UID: \"b9c106a2-ce74-4c57-80c1-b1f75da0d43d\") " pod="openshift-insights/insights-operator-7c4986d449-964q9" Nov 03 16:14:17.727406 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.727389 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtbz7\" (UniqueName: \"kubernetes.io/projected/bf8c316f-51ce-4c69-865f-1d481ff30630-kube-api-access-gtbz7\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4\" (UID: \"bf8c316f-51ce-4c69-865f-1d481ff30630\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" Nov 03 16:14:17.727595 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.727578 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svnps\" (UniqueName: \"kubernetes.io/projected/c2ded168-1814-4922-9eee-00caf304b8af-kube-api-access-svnps\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:17.727826 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.727810 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdzb4\" (UniqueName: \"kubernetes.io/projected/8f85dfa8-9158-437d-b151-d8eb8fe65b7b-kube-api-access-xdzb4\") pod \"ingress-canary-jdthd\" (UID: \"8f85dfa8-9158-437d-b151-d8eb8fe65b7b\") " pod="openshift-ingress-canary/ingress-canary-jdthd" Nov 03 16:14:17.731415 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.731391 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-bound-sa-token\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:17.732237 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.732218 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfkpb\" (UniqueName: \"kubernetes.io/projected/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-api-access-kfkpb\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:17.745137 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.745101 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzhxd\" (UniqueName: \"kubernetes.io/projected/4b370319-6de8-41e2-ae5f-21267a2fba74-kube-api-access-pzhxd\") pod \"kas-connection-checker-685895c7c9-hxr8b\" (UID: \"4b370319-6de8-41e2-ae5f-21267a2fba74\") " pod="kube-system/kas-connection-checker-685895c7c9-hxr8b" Nov 03 16:14:17.745262 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.745213 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d2cr\" (UniqueName: \"kubernetes.io/projected/94fb68fc-92af-4e72-824b-7092f9875560-kube-api-access-2d2cr\") pod \"migrator-66df5d496b-g8vzv\" (UID: \"94fb68fc-92af-4e72-824b-7092f9875560\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-g8vzv" Nov 03 16:14:17.745314 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.745267 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4dbx\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-kube-api-access-z4dbx\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:17.753636 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.753124 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-bound-sa-token\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:17.753636 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.753190 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp88d\" (UniqueName: \"kubernetes.io/projected/7731b228-5421-4121-9228-1f19a850ef9a-kube-api-access-vp88d\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.753636 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.753241 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwcf4\" (UniqueName: \"kubernetes.io/projected/1566c753-13b1-4495-9268-05e3676d82bd-kube-api-access-vwcf4\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:14:17.753636 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.753244 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn2hk\" (UniqueName: \"kubernetes.io/projected/812067ca-eef5-4ab1-a24e-61796ac0721b-kube-api-access-hn2hk\") pod \"node-ca-bflh2\" (UID: \"812067ca-eef5-4ab1-a24e-61796ac0721b\") " pod="openshift-image-registry/node-ca-bflh2" Nov 03 16:14:17.753636 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.753323 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4h7b\" (UniqueName: \"kubernetes.io/projected/3cbd4c91-69f0-47be-94be-e451d44b4992-kube-api-access-l4h7b\") pod \"node-resolver-zfshj\" (UID: \"3cbd4c91-69f0-47be-94be-e451d44b4992\") " pod="openshift-dns/node-resolver-zfshj" Nov 03 16:14:17.753636 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.753343 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phmbx\" (UniqueName: \"kubernetes.io/projected/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-kube-api-access-phmbx\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:17.753636 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.753358 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz7ht\" (UniqueName: \"kubernetes.io/projected/216c0d66-a3ec-4748-b761-3420536bfa3a-kube-api-access-wz7ht\") pod \"console-operator-7cd56b8dbc-tnh8w\" (UID: \"216c0d66-a3ec-4748-b761-3420536bfa3a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:17.753636 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.753592 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75xq7\" (UniqueName: \"kubernetes.io/projected/66b43213-9ec2-4164-9f8d-d7621bd57eb3-kube-api-access-75xq7\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:17.754008 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.753687 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fx4n\" (UniqueName: \"kubernetes.io/projected/1580792e-eb86-4d79-92a7-8055e6a8aaff-kube-api-access-8fx4n\") pod \"service-ca-operator-544d749c8c-qqcft\" (UID: \"1580792e-eb86-4d79-92a7-8055e6a8aaff\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" Nov 03 16:14:17.754008 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.753691 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42fxm\" (UniqueName: \"kubernetes.io/projected/4cbc83f2-64b1-4804-a180-43df6459ffa4-kube-api-access-42fxm\") pod \"kas-connection-checker-685895c7c9-4j5k5\" (UID: \"4cbc83f2-64b1-4804-a180-43df6459ffa4\") " pod="kube-system/kas-connection-checker-685895c7c9-4j5k5" Nov 03 16:14:17.754008 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.753761 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9hhm\" (UniqueName: \"kubernetes.io/projected/a7f5be2e-c57e-432f-a5bb-144709f3b492-kube-api-access-j9hhm\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.754008 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.753809 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g2jb\" (UniqueName: \"kubernetes.io/projected/78e9f545-8400-48b6-968d-eb65dc94b471-kube-api-access-9g2jb\") pod \"kas-connection-checker-685895c7c9-j8t6k\" (UID: \"78e9f545-8400-48b6-968d-eb65dc94b471\") " pod="kube-system/kas-connection-checker-685895c7c9-j8t6k" Nov 03 16:14:17.755947 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.755928 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z647\" (UniqueName: \"kubernetes.io/projected/55767581-2059-49bf-8dd0-69e31d5adb83-kube-api-access-2z647\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:14:17.756060 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.756041 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfv4w\" (UniqueName: \"kubernetes.io/projected/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-kube-api-access-cfv4w\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.756122 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.756048 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjwvk\" (UniqueName: \"kubernetes.io/projected/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-kube-api-access-gjwvk\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.758236 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.758219 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vdm\" (UniqueName: \"kubernetes.io/projected/fa01a744-9121-4f6a-b280-dba215c1ab8f-kube-api-access-d4vdm\") pod \"volume-data-source-validator-58c9f5f46d-88dnt\" (UID: \"fa01a744-9121-4f6a-b280-dba215c1ab8f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-88dnt" Nov 03 16:14:17.761408 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.761361 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tszlg\" (UniqueName: \"kubernetes.io/projected/ba6c03a7-1ba9-48c0-b67b-6fb26263425e-kube-api-access-tszlg\") pod \"network-check-source-6f78d59f99-cm2k2\" (UID: \"ba6c03a7-1ba9-48c0-b67b-6fb26263425e\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-cm2k2" Nov 03 16:14:17.763831 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.763798 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9slx\" (UniqueName: \"kubernetes.io/projected/110bff76-1581-4379-8468-441a3a81dea2-kube-api-access-l9slx\") pod \"network-check-target-5fq8q\" (UID: \"110bff76-1581-4379-8468-441a3a81dea2\") " pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:14:17.764085 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.764070 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jjxv\" (UniqueName: \"kubernetes.io/projected/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd-kube-api-access-5jjxv\") pod \"iptables-alerter-g5w4j\" (UID: \"ec0f2c68-a3bf-4475-ad0d-43346dbd32fd\") " pod="openshift-network-operator/iptables-alerter-g5w4j" Nov 03 16:14:17.795464 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795422 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-slash\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.795464 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795466 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1566c753-13b1-4495-9268-05e3676d82bd-system-cni-dir\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:14:17.795739 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795491 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-kubelet\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.795739 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795515 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-lib-modules\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.795739 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795538 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-systemd\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.795739 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795559 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c96f345-2cda-4554-bef3-be416dc50225-dbus\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:14:17.795739 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795555 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-slash\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.795739 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795579 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-run-netns\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.795739 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795610 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-etc-openvswitch\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.795739 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795618 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-run-netns\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.795739 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795623 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c96f345-2cda-4554-bef3-be416dc50225-dbus\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:14:17.795739 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795641 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.795739 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795661 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-etc-openvswitch\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.795739 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795566 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-kubelet\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.795739 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795623 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-systemd\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.795739 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795555 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1566c753-13b1-4495-9268-05e3676d82bd-system-cni-dir\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:14:17.795739 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795661 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-lib-modules\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.795739 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795663 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795754 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1566c753-13b1-4495-9268-05e3676d82bd-cnibin\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795755 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795774 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-system-cni-dir\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.795776 4947 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795807 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-multus-socket-dir-parent\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795813 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1566c753-13b1-4495-9268-05e3676d82bd-cnibin\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795833 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-etc-kubernetes\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795856 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-sys\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795862 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-system-cni-dir\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.795873 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls podName:c2ded168-1814-4922-9eee-00caf304b8af nodeName:}" failed. No retries permitted until 2026-11-03 16:14:18.29585602 +0000 UTC m=+1.884855967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vmcs4" (UID: "c2ded168-1814-4922-9eee-00caf304b8af") : secret "samples-operator-tls" not found Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795893 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-run-systemd\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795897 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-sys\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795901 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-etc-kubernetes\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795897 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-multus-socket-dir-parent\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795924 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795941 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-run-systemd\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795953 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-etc-selinux\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.795979 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-hostroot\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.795988 4947 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796006 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-kubernetes\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.796025 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls podName:f02c434e-ede0-4bea-8830-f3444c21a6a6 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:18.296013594 +0000 UTC m=+1.885013544 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rj5rd" (UID: "f02c434e-ede0-4bea-8830-f3444c21a6a6") : secret "insights-runtime-extractor-tls" not found Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796051 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-kubernetes\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796051 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796074 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-hostroot\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796108 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-socket-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796061 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-etc-selinux\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796132 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796166 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-run-ovn-kubernetes\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796173 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-cnibin\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796197 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-socket-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796209 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-cnibin\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796224 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-var-lib-openvswitch\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796246 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:17.796275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796283 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-multus-cni-dir\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796306 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cbd4c91-69f0-47be-94be-e451d44b4992-hosts-file\") pod \"node-resolver-zfshj\" (UID: \"3cbd4c91-69f0-47be-94be-e451d44b4992\") " pod="openshift-dns/node-resolver-zfshj" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796314 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-var-lib-openvswitch\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796331 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.796359 4947 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796373 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-multus-cni-dir\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796409 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3cbd4c91-69f0-47be-94be-e451d44b4992-hosts-file\") pod \"node-resolver-zfshj\" (UID: \"3cbd4c91-69f0-47be-94be-e451d44b4992\") " pod="openshift-dns/node-resolver-zfshj" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796362 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-sysctl-d\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.796415 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls podName:5ced7a93-d9f0-44de-9a76-34a84e41843b nodeName:}" failed. No retries permitted until 2026-11-03 16:14:18.296402956 +0000 UTC m=+1.885402920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls") pod "dns-default-xfk72" (UID: "5ced7a93-d9f0-44de-9a76-34a84e41843b") : secret "dns-default-metrics-tls" not found Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.796459 4947 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796499 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1566c753-13b1-4495-9268-05e3676d82bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796462 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-sysctl-d\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796458 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1566c753-13b1-4495-9268-05e3676d82bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r6mtg\" (UID: \"1566c753-13b1-4495-9268-05e3676d82bd\") " pod="openshift-multus/multus-additional-cni-plugins-r6mtg" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.796519 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs podName:66b43213-9ec2-4164-9f8d-d7621bd57eb3 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:18.296503109 +0000 UTC m=+1.885503070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs") pod "router-default-5dc56589f5-wsj6w" (UID: "66b43213-9ec2-4164-9f8d-d7621bd57eb3") : secret "router-metrics-certs-default" not found Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796556 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-run-ovn\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796574 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f02c434e-ede0-4bea-8830-f3444c21a6a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796584 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-run-k8s-cni-cncf-io\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796608 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-var-lib-kubelet\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796622 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-run-k8s-cni-cncf-io\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796646 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-run-ovn\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796647 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-registration-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796649 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-var-lib-kubelet\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796714 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-log-socket\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796728 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-registration-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796747 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796762 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-host\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796787 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-node-log\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796818 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-sys-fs\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796821 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-host\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796842 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-cni-bin\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796858 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-log-socket\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796868 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-run-netns\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796896 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-node-log\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796910 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-run-netns\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796915 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-sys-fs\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.796903 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle podName:66b43213-9ec2-4164-9f8d-d7621bd57eb3 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:18.296885809 +0000 UTC m=+1.885885763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle") pod "router-default-5dc56589f5-wsj6w" (UID: "66b43213-9ec2-4164-9f8d-d7621bd57eb3") : configmap references non-existent config key: service-ca.crt Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796896 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-cni-bin\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796939 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-sysctl-conf\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796960 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r755z\" (UID: \"2c812633-b15d-4a30-982d-80d276ec78c7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796978 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-sysconfig\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.796996 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-run\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797013 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-sysctl-conf\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797019 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/812067ca-eef5-4ab1-a24e-61796ac0721b-host\") pod \"node-ca-bflh2\" (UID: \"812067ca-eef5-4ab1-a24e-61796ac0721b\") " pod="openshift-image-registry/node-ca-bflh2" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797038 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-sysconfig\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797045 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-var-lib-cni-bin\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797051 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-run\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.797054 4947 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797073 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797076 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/812067ca-eef5-4ab1-a24e-61796ac0721b-host\") pod \"node-ca-bflh2\" (UID: \"812067ca-eef5-4ab1-a24e-61796ac0721b\") " pod="openshift-image-registry/node-ca-bflh2" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.797094 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert podName:2c812633-b15d-4a30-982d-80d276ec78c7 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:18.297081795 +0000 UTC m=+1.886081769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r755z" (UID: "2c812633-b15d-4a30-982d-80d276ec78c7") : secret "networking-console-plugin-cert" not found Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.797127 4947 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797126 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-var-lib-cni-bin\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.797179 4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797125 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.797189 4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d945bd975-wtl6x: secret "image-registry-tls" not found Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.797194 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs podName:55767581-2059-49bf-8dd0-69e31d5adb83 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:18.297183236 +0000 UTC m=+1.886183226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs") pod "network-metrics-daemon-n7hw7" (UID: "55767581-2059-49bf-8dd0-69e31d5adb83") : secret "metrics-daemon-secret" not found Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.797259 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls podName:03cc593d-32bc-4453-8a45-2b546f862acf nodeName:}" failed. No retries permitted until 2026-11-03 16:14:18.297246293 +0000 UTC m=+1.886246247 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls") pod "image-registry-7d945bd975-wtl6x" (UID: "03cc593d-32bc-4453-8a45-2b546f862acf") : secret "image-registry-tls" not found Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797298 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-device-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797327 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-cni-netd\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797353 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-modprobe-d\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797378 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f02c434e-ede0-4bea-8830-f3444c21a6a6-crio-socket\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797385 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a3ce380e-3ddd-42bd-867c-9bba84f2bab4-device-dir\") pod \"aws-ebs-csi-driver-node-gjq8g\" (UID: \"a3ce380e-3ddd-42bd-867c-9bba84f2bab4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjq8g" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797402 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-cni-netd\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797409 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-systemd-units\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797428 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f02c434e-ede0-4bea-8830-f3444c21a6a6-crio-socket\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797446 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-systemd-units\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797446 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-etc-modprobe-d\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797446 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-var-lib-cni-multus\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797500 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-run-multus-certs\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797519 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c96f345-2cda-4554-bef3-be416dc50225-kubelet-config\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797524 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-var-lib-cni-multus\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797539 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797557 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-host-run-multus-certs\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797566 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c96f345-2cda-4554-bef3-be416dc50225-kubelet-config\") pod \"global-pull-secret-syncer-p7vvq\" (UID: \"8c96f345-2cda-4554-bef3-be416dc50225\") " pod="kube-system/global-pull-secret-syncer-p7vvq" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797557 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.797604 4947 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.797619 4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797623 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-run-openvswitch\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.797573 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.797627 4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68c8fc97c8-vxzz9: secret "image-registry-tls" not found Nov 03 16:14:17.799708 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.797645 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls podName:e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:18.297635559 +0000 UTC m=+1.886635507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bw6bt" (UID: "e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28") : secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:17.799708 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797602 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-run-openvswitch\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.799708 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:17.797659 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls podName:a13e4eb2-b4f2-4f66-85d3-8bedb078823c nodeName:}" failed. No retries permitted until 2026-11-03 16:14:18.297650364 +0000 UTC m=+1.886650348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls") pod "image-registry-68c8fc97c8-vxzz9" (UID: "a13e4eb2-b4f2-4f66-85d3-8bedb078823c") : secret "image-registry-tls" not found Nov 03 16:14:17.799708 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797679 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.799708 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797720 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd-host-slash\") pod \"iptables-alerter-g5w4j\" (UID: \"ec0f2c68-a3bf-4475-ad0d-43346dbd32fd\") " pod="openshift-network-operator/iptables-alerter-g5w4j" Nov 03 16:14:17.799708 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797737 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-var-lib-kubelet\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.799708 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797752 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-multus-conf-dir\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.799708 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797799 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ec0f2c68-a3bf-4475-ad0d-43346dbd32fd-host-slash\") pod \"iptables-alerter-g5w4j\" (UID: \"ec0f2c68-a3bf-4475-ad0d-43346dbd32fd\") " pod="openshift-network-operator/iptables-alerter-g5w4j" Nov 03 16:14:17.799708 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797804 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7731b228-5421-4121-9228-1f19a850ef9a-multus-conf-dir\") pod \"multus-kmfdd\" (UID: \"7731b228-5421-4121-9228-1f19a850ef9a\") " pod="openshift-multus/multus-kmfdd" Nov 03 16:14:17.799708 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797815 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ec02fd2-78a5-4813-b6e4-4edefa217fe1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gvrm4\" (UID: \"7ec02fd2-78a5-4813-b6e4-4edefa217fe1\") " pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:17.799708 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.797824 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f5be2e-c57e-432f-a5bb-144709f3b492-var-lib-kubelet\") pod \"tuned-2kl7n\" (UID: \"a7f5be2e-c57e-432f-a5bb-144709f3b492\") " pod="openshift-cluster-node-tuning-operator/tuned-2kl7n" Nov 03 16:14:17.860481 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:17.860438 4947 scope.go:122] "RemoveContainer" containerID="02633c14002a1b44b88bd7bf4802aa104ef3c6c904de369370b2ccfa72bbf740" Nov 03 16:14:18.302413 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:18.302317 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:18.302413 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:18.302361 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:18.302413 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:18.302386 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r755z\" (UID: \"2c812633-b15d-4a30-982d-80d276ec78c7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:18.302413 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:18.302408 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:18.302424 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:18.302464 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302479 4947 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302500 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle podName:66b43213-9ec2-4164-9f8d-d7621bd57eb3 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:19.302479902 +0000 UTC m=+2.891479864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle") pod "router-default-5dc56589f5-wsj6w" (UID: "66b43213-9ec2-4164-9f8d-d7621bd57eb3") : configmap references non-existent config key: service-ca.crt Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302510 4947 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302530 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs podName:66b43213-9ec2-4164-9f8d-d7621bd57eb3 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:19.302518652 +0000 UTC m=+2.891518616 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs") pod "router-default-5dc56589f5-wsj6w" (UID: "66b43213-9ec2-4164-9f8d-d7621bd57eb3") : secret "router-metrics-certs-default" not found Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:18.302530 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302550 4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302564 4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68c8fc97c8-vxzz9: secret "image-registry-tls" not found Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302510 4947 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302587 4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302566 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs podName:55767581-2059-49bf-8dd0-69e31d5adb83 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:19.302550156 +0000 UTC m=+2.891550129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs") pod "network-metrics-daemon-n7hw7" (UID: "55767581-2059-49bf-8dd0-69e31d5adb83") : secret "metrics-daemon-secret" not found Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302599 4947 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302603 4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d945bd975-wtl6x: secret "image-registry-tls" not found Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302623 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert podName:2c812633-b15d-4a30-982d-80d276ec78c7 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:19.302610148 +0000 UTC m=+2.891610132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r755z" (UID: "2c812633-b15d-4a30-982d-80d276ec78c7") : secret "networking-console-plugin-cert" not found Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302649 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls podName:a13e4eb2-b4f2-4f66-85d3-8bedb078823c nodeName:}" failed. No retries permitted until 2026-11-03 16:14:19.302640785 +0000 UTC m=+2.891640735 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls") pod "image-registry-68c8fc97c8-vxzz9" (UID: "a13e4eb2-b4f2-4f66-85d3-8bedb078823c") : secret "image-registry-tls" not found Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302661 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls podName:e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:19.302656106 +0000 UTC m=+2.891656052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bw6bt" (UID: "e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28") : secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302673 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls podName:03cc593d-32bc-4453-8a45-2b546f862acf nodeName:}" failed. No retries permitted until 2026-11-03 16:14:19.302666495 +0000 UTC m=+2.891666442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls") pod "image-registry-7d945bd975-wtl6x" (UID: "03cc593d-32bc-4453-8a45-2b546f862acf") : secret "image-registry-tls" not found Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:18.302733 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:18.302759 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:18.302764 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:18.303337 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:18.302798 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:18.303337 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302864 4947 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 16:14:18.303337 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302870 4947 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 16:14:18.303337 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302882 4947 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 16:14:18.303337 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302905 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls podName:c2ded168-1814-4922-9eee-00caf304b8af nodeName:}" failed. No retries permitted until 2026-11-03 16:14:19.302894117 +0000 UTC m=+2.891894068 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vmcs4" (UID: "c2ded168-1814-4922-9eee-00caf304b8af") : secret "samples-operator-tls" not found Nov 03 16:14:18.303337 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302920 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls podName:5ced7a93-d9f0-44de-9a76-34a84e41843b nodeName:}" failed. No retries permitted until 2026-11-03 16:14:19.302911071 +0000 UTC m=+2.891911047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls") pod "dns-default-xfk72" (UID: "5ced7a93-d9f0-44de-9a76-34a84e41843b") : secret "dns-default-metrics-tls" not found Nov 03 16:14:18.303337 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.302935 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls podName:f02c434e-ede0-4bea-8830-f3444c21a6a6 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:19.302929604 +0000 UTC m=+2.891929550 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rj5rd" (UID: "f02c434e-ede0-4bea-8830-f3444c21a6a6") : secret "insights-runtime-extractor-tls" not found Nov 03 16:14:18.623022 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:18.622946 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:14:18.623410 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:18.623319 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/1.log" Nov 03 16:14:18.623410 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:18.623347 4947 generic.go:356] "Generic (PLEG): container finished" podID="216c0d66-a3ec-4748-b761-3420536bfa3a" containerID="b28d6b7ec12d483811bb785fad726a2c5a2c327c8c52ad567df729285fe95977" exitCode=255 Nov 03 16:14:18.623493 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:18.623425 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" event={"ID":"216c0d66-a3ec-4748-b761-3420536bfa3a","Type":"ContainerDied","Data":"b28d6b7ec12d483811bb785fad726a2c5a2c327c8c52ad567df729285fe95977"} Nov 03 16:14:18.623536 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:18.623507 4947 scope.go:122] "RemoveContainer" containerID="02633c14002a1b44b88bd7bf4802aa104ef3c6c904de369370b2ccfa72bbf740" Nov 03 16:14:18.623967 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:18.623944 4947 scope.go:122] "RemoveContainer" containerID="b28d6b7ec12d483811bb785fad726a2c5a2c327c8c52ad567df729285fe95977" Nov 03 16:14:18.624174 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:18.624153 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tnh8w_openshift-console-operator(216c0d66-a3ec-4748-b761-3420536bfa3a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" podUID="216c0d66-a3ec-4748-b761-3420536bfa3a" Nov 03 16:14:19.313233 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:19.313194 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:19.313233 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:19.313237 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:19.313447 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:19.313264 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:19.313447 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:19.313293 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:19.313447 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:19.313319 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:19.313447 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313343 4947 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 16:14:19.313447 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:19.313349 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r755z\" (UID: \"2c812633-b15d-4a30-982d-80d276ec78c7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:19.313447 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:19.313386 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:14:19.313447 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313343 4947 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 16:14:19.313447 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313425 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls podName:f02c434e-ede0-4bea-8830-f3444c21a6a6 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.313405493 +0000 UTC m=+4.902405463 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rj5rd" (UID: "f02c434e-ede0-4bea-8830-f3444c21a6a6") : secret "insights-runtime-extractor-tls" not found Nov 03 16:14:19.313447 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313425 4947 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 16:14:19.313447 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313441 4947 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 16:14:19.313846 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313455 4947 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 16:14:19.313846 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313445 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle podName:66b43213-9ec2-4164-9f8d-d7621bd57eb3 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.313435761 +0000 UTC m=+4.902435710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle") pod "router-default-5dc56589f5-wsj6w" (UID: "66b43213-9ec2-4164-9f8d-d7621bd57eb3") : configmap references non-existent config key: service-ca.crt Nov 03 16:14:19.313846 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313500 4947 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 16:14:19.313846 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313511 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs podName:66b43213-9ec2-4164-9f8d-d7621bd57eb3 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.313495293 +0000 UTC m=+4.902495240 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs") pod "router-default-5dc56589f5-wsj6w" (UID: "66b43213-9ec2-4164-9f8d-d7621bd57eb3") : secret "router-metrics-certs-default" not found Nov 03 16:14:19.313846 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313535 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls podName:c2ded168-1814-4922-9eee-00caf304b8af nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.313524833 +0000 UTC m=+4.902524788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-vmcs4" (UID: "c2ded168-1814-4922-9eee-00caf304b8af") : secret "samples-operator-tls" not found Nov 03 16:14:19.313846 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313549 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls podName:5ced7a93-d9f0-44de-9a76-34a84e41843b nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.313540816 +0000 UTC m=+4.902540763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls") pod "dns-default-xfk72" (UID: "5ced7a93-d9f0-44de-9a76-34a84e41843b") : secret "dns-default-metrics-tls" not found Nov 03 16:14:19.313846 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313552 4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:19.313846 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:19.313505 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:19.313846 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313563 4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d945bd975-wtl6x: secret "image-registry-tls" not found Nov 03 16:14:19.313846 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313571 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert podName:2c812633-b15d-4a30-982d-80d276ec78c7 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.313554066 +0000 UTC m=+4.902554034 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r755z" (UID: "2c812633-b15d-4a30-982d-80d276ec78c7") : secret "networking-console-plugin-cert" not found Nov 03 16:14:19.313846 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313648 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs podName:55767581-2059-49bf-8dd0-69e31d5adb83 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.313632144 +0000 UTC m=+4.902632106 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs") pod "network-metrics-daemon-n7hw7" (UID: "55767581-2059-49bf-8dd0-69e31d5adb83") : secret "metrics-daemon-secret" not found Nov 03 16:14:19.313846 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313663 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls podName:03cc593d-32bc-4453-8a45-2b546f862acf nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.313657136 +0000 UTC m=+4.902657082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls") pod "image-registry-7d945bd975-wtl6x" (UID: "03cc593d-32bc-4453-8a45-2b546f862acf") : secret "image-registry-tls" not found Nov 03 16:14:19.313846 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:19.313740 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:19.313846 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:19.313765 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:19.313846 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313820 4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:19.313846 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313831 4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68c8fc97c8-vxzz9: secret "image-registry-tls" not found Nov 03 16:14:19.313846 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313852 4947 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:19.314315 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313860 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls podName:a13e4eb2-b4f2-4f66-85d3-8bedb078823c nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.31385125 +0000 UTC m=+4.902851231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls") pod "image-registry-68c8fc97c8-vxzz9" (UID: "a13e4eb2-b4f2-4f66-85d3-8bedb078823c") : secret "image-registry-tls" not found Nov 03 16:14:19.314315 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:19.313891 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls podName:e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:21.313882965 +0000 UTC m=+4.902882927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bw6bt" (UID: "e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28") : secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:19.502450 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:19.502420 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zfshj_3cbd4c91-69f0-47be-94be-e451d44b4992/dns-node-resolver/0.log" Nov 03 16:14:19.629152 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:19.629077 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:14:19.712073 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:19.712048 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:20.184676 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:20.184638 4947 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:20.185017 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:20.185004 4947 scope.go:122] "RemoveContainer" containerID="b28d6b7ec12d483811bb785fad726a2c5a2c327c8c52ad567df729285fe95977" Nov 03 16:14:20.185202 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:20.185185 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tnh8w_openshift-console-operator(216c0d66-a3ec-4748-b761-3420536bfa3a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" podUID="216c0d66-a3ec-4748-b761-3420536bfa3a" Nov 03 16:14:20.464406 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:20.464329 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bflh2_812067ca-eef5-4ab1-a24e-61796ac0721b/node-ca/0.log" Nov 03 16:14:21.064315 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.064282 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jdthd_8f85dfa8-9158-437d-b151-d8eb8fe65b7b/serve-healthcheck-canary/0.log" Nov 03 16:14:21.264491 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.264461 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-g8vzv_94fb68fc-92af-4e72-824b-7092f9875560/migrator/0.log" Nov 03 16:14:21.333326 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.333226 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:21.333326 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.333281 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:21.333326 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.333315 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:21.333613 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.333344 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:21.333613 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.333378 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:21.333613 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.333410 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r755z\" (UID: \"2c812633-b15d-4a30-982d-80d276ec78c7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:21.333613 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:21.333446 4947 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 16:14:21.333613 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:21.333489 4947 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 16:14:21.333613 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:21.333489 4947 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 16:14:21.333613 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:21.333514 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls podName:f02c434e-ede0-4bea-8830-f3444c21a6a6 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:25.333494978 +0000 UTC m=+8.922494926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rj5rd" (UID: "f02c434e-ede0-4bea-8830-f3444c21a6a6") : secret "insights-runtime-extractor-tls" not found Nov 03 16:14:21.333613 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:21.333489 4947 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 16:14:21.333613 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:21.333539 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert podName:2c812633-b15d-4a30-982d-80d276ec78c7 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:25.333523983 +0000 UTC m=+8.922523943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-r755z" (UID: "2c812633-b15d-4a30-982d-80d276ec78c7") : secret "networking-console-plugin-cert" not found Nov 03 16:14:21.333613 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:21.333558 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls podName:5ced7a93-d9f0-44de-9a76-34a84e41843b nodeName:}" failed. No retries permitted until 2026-11-03 16:14:25.333547507 +0000 UTC m=+8.922547454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls") pod "dns-default-xfk72" (UID: "5ced7a93-d9f0-44de-9a76-34a84e41843b") : secret "dns-default-metrics-tls" not found Nov 03 16:14:21.333613 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.333579 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:14:21.333613 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.333611 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:21.334117 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:21.333642 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs podName:66b43213-9ec2-4164-9f8d-d7621bd57eb3 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:25.333630581 +0000 UTC m=+8.922630543 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs") pod "router-default-5dc56589f5-wsj6w" (UID: "66b43213-9ec2-4164-9f8d-d7621bd57eb3") : secret "router-metrics-certs-default" not found Nov 03 16:14:21.334117 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:21.333687 4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:21.334117 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.333684 4947 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 16:14:21.334117 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.333723 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:21.334117 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.333756 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:21.334117 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:21.333722 4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d945bd975-wtl6x: secret "image-registry-tls" not found Nov 03 16:14:21.334117 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:21.333860 4947 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:21.334117 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:21.333903 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls podName:e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:25.333891144 +0000 UTC m=+8.922891105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bw6bt" (UID: "e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28") : secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:21.334117 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:21.333833 4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:21.334117 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:21.333919 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls podName:03cc593d-32bc-4453-8a45-2b546f862acf nodeName:}" failed. No retries permitted until 2026-11-03 16:14:25.333910552 +0000 UTC m=+8.922910538 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls") pod "image-registry-7d945bd975-wtl6x" (UID: "03cc593d-32bc-4453-8a45-2b546f862acf") : secret "image-registry-tls" not found Nov 03 16:14:21.334117 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:21.333923 4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68c8fc97c8-vxzz9: secret "image-registry-tls" not found Nov 03 16:14:21.334117 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:21.333960 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls podName:a13e4eb2-b4f2-4f66-85d3-8bedb078823c nodeName:}" failed. No retries permitted until 2026-11-03 16:14:25.333949293 +0000 UTC m=+8.922949239 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls") pod "image-registry-68c8fc97c8-vxzz9" (UID: "a13e4eb2-b4f2-4f66-85d3-8bedb078823c") : secret "image-registry-tls" not found Nov 03 16:14:21.334117 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.334096 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66b43213-9ec2-4164-9f8d-d7621bd57eb3-service-ca-bundle\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:21.336039 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.336017 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55767581-2059-49bf-8dd0-69e31d5adb83-metrics-certs\") pod \"network-metrics-daemon-n7hw7\" (UID: \"55767581-2059-49bf-8dd0-69e31d5adb83\") " pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:14:21.336170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.336151 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c2ded168-1814-4922-9eee-00caf304b8af-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-vmcs4\" (UID: \"c2ded168-1814-4922-9eee-00caf304b8af\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:21.460074 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.460031 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" Nov 03 16:14:21.463563 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.463543 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n7hw7" Nov 03 16:14:21.665441 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.665366 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4_bf8c316f-51ce-4c69-865f-1d481ff30630/kube-storage-version-migrator-operator/0.log" Nov 03 16:14:21.775590 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.775552 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n7hw7"] Nov 03 16:14:21.777727 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:21.777678 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55767581_2059_49bf_8dd0_69e31d5adb83.slice/crio-e9315dead78378873f50cd5d80fc6fa15beee2c5b44f80253dce457ceb200f38 WatchSource:0}: Error finding container e9315dead78378873f50cd5d80fc6fa15beee2c5b44f80253dce457ceb200f38: Status 404 returned error can't find the container with id e9315dead78378873f50cd5d80fc6fa15beee2c5b44f80253dce457ceb200f38 Nov 03 16:14:21.779714 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.779676 4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:14:21.789827 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:21.789802 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4"] Nov 03 16:14:22.604092 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:22.602465 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:22.604092 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:22.602726 4947 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 16:14:22.643869 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:22.643829 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" event={"ID":"c2ded168-1814-4922-9eee-00caf304b8af","Type":"ContainerStarted","Data":"61ab7864c307840aa8cebe324dd68d6716825581df27bb5e21ed12f2e182c22b"} Nov 03 16:14:22.645323 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:22.645293 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n7hw7" event={"ID":"55767581-2059-49bf-8dd0-69e31d5adb83","Type":"ContainerStarted","Data":"e9315dead78378873f50cd5d80fc6fa15beee2c5b44f80253dce457ceb200f38"} Nov 03 16:14:23.651195 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:23.651161 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" event={"ID":"c2ded168-1814-4922-9eee-00caf304b8af","Type":"ContainerStarted","Data":"116086aed01020508a6fa39859bb2b3a6bd153e9f12aa52bd79639aa4813ac45"} Nov 03 16:14:23.651644 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:23.651225 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-vmcs4" event={"ID":"c2ded168-1814-4922-9eee-00caf304b8af","Type":"ContainerStarted","Data":"84c3c3031396b3c8d91b21152c584c9baa42e85069546e7f3d7f766748faaa10"} Nov 03 16:14:23.652980 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:23.652956 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n7hw7" event={"ID":"55767581-2059-49bf-8dd0-69e31d5adb83","Type":"ContainerStarted","Data":"4c36abbd9866e3eee9b052595e1f8f4e497eb8111f500236ba7b2f533d9b7321"} Nov 03 16:14:23.653109 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:23.652998 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n7hw7" event={"ID":"55767581-2059-49bf-8dd0-69e31d5adb83","Type":"ContainerStarted","Data":"4560b707243520b03362efbf0de39b4cf2cdd9ac4ac93c3da527134f1beaea64"} Nov 03 16:14:24.973674 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:24.973635 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:24.974057 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:24.974036 4947 scope.go:122] "RemoveContainer" containerID="b28d6b7ec12d483811bb785fad726a2c5a2c327c8c52ad567df729285fe95977" Nov 03 16:14:24.974250 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:24.974221 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tnh8w_openshift-console-operator(216c0d66-a3ec-4748-b761-3420536bfa3a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" podUID="216c0d66-a3ec-4748-b761-3420536bfa3a" Nov 03 16:14:25.373257 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:25.373219 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:25.373257 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:25.373260 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:25.373502 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:25.373300 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:25.373502 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:25.373328 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:25.373502 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:25.373350 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:25.373502 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:25.373377 4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:25.373502 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:25.373388 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r755z\" (UID: \"2c812633-b15d-4a30-982d-80d276ec78c7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:25.373502 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:25.373398 4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68c8fc97c8-vxzz9: secret "image-registry-tls" not found Nov 03 16:14:25.373502 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:25.373421 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:25.373502 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:25.373436 4947 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 16:14:25.373502 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:25.373455 4947 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:25.373502 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:25.373465 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls podName:a13e4eb2-b4f2-4f66-85d3-8bedb078823c nodeName:}" failed. No retries permitted until 2026-11-03 16:14:33.373445763 +0000 UTC m=+16.962445727 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls") pod "image-registry-68c8fc97c8-vxzz9" (UID: "a13e4eb2-b4f2-4f66-85d3-8bedb078823c") : secret "image-registry-tls" not found Nov 03 16:14:25.373502 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:25.373501 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls podName:f02c434e-ede0-4bea-8830-f3444c21a6a6 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:33.37348352 +0000 UTC m=+16.962483484 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rj5rd" (UID: "f02c434e-ede0-4bea-8830-f3444c21a6a6") : secret "insights-runtime-extractor-tls" not found Nov 03 16:14:25.373502 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:25.373502 4947 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 16:14:25.374020 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:25.373512 4947 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 16:14:25.374020 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:25.373521 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls podName:e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:33.37351224 +0000 UTC m=+16.962512190 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-bw6bt" (UID: "e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28") : secret "cluster-monitoring-operator-tls" not found Nov 03 16:14:25.374020 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:25.373527 4947 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7d945bd975-wtl6x: secret "image-registry-tls" not found Nov 03 16:14:25.374020 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:25.373548 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs podName:66b43213-9ec2-4164-9f8d-d7621bd57eb3 nodeName:}" failed. No retries permitted until 2026-11-03 16:14:33.373536147 +0000 UTC m=+16.962536117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs") pod "router-default-5dc56589f5-wsj6w" (UID: "66b43213-9ec2-4164-9f8d-d7621bd57eb3") : secret "router-metrics-certs-default" not found Nov 03 16:14:25.374020 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:14:25.373565 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls podName:03cc593d-32bc-4453-8a45-2b546f862acf nodeName:}" failed. No retries permitted until 2026-11-03 16:14:33.373555836 +0000 UTC m=+16.962555783 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls") pod "image-registry-7d945bd975-wtl6x" (UID: "03cc593d-32bc-4453-8a45-2b546f862acf") : secret "image-registry-tls" not found Nov 03 16:14:25.376031 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:25.376013 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5ced7a93-d9f0-44de-9a76-34a84e41843b-metrics-tls\") pod \"dns-default-xfk72\" (UID: \"5ced7a93-d9f0-44de-9a76-34a84e41843b\") " pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:25.376111 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:25.376092 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c812633-b15d-4a30-982d-80d276ec78c7-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-r755z\" (UID: \"2c812633-b15d-4a30-982d-80d276ec78c7\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:25.662143 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:25.662061 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" Nov 03 16:14:25.669212 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:25.669180 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:25.966043 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:25.966021 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-r755z"] Nov 03 16:14:25.967647 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:25.967617 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c812633_b15d_4a30_982d_80d276ec78c7.slice/crio-e0267110fa5785583c5bc27af80c5d6b1118f900db5d3576a7f80d4b17d8ac40 WatchSource:0}: Error finding container e0267110fa5785583c5bc27af80c5d6b1118f900db5d3576a7f80d4b17d8ac40: Status 404 returned error can't find the container with id e0267110fa5785583c5bc27af80c5d6b1118f900db5d3576a7f80d4b17d8ac40 Nov 03 16:14:25.987777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:25.987748 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xfk72"] Nov 03 16:14:25.990412 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:25.990384 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ced7a93_d9f0_44de_9a76_34a84e41843b.slice/crio-53535c380c788bf17bf867f37f78324249c0ce192e922f3948c2006af28365fa WatchSource:0}: Error finding container 53535c380c788bf17bf867f37f78324249c0ce192e922f3948c2006af28365fa: Status 404 returned error can't find the container with id 53535c380c788bf17bf867f37f78324249c0ce192e922f3948c2006af28365fa Nov 03 16:14:26.536733 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:26.536681 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:14:26.663317 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:26.663281 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xfk72" event={"ID":"5ced7a93-d9f0-44de-9a76-34a84e41843b","Type":"ContainerStarted","Data":"53535c380c788bf17bf867f37f78324249c0ce192e922f3948c2006af28365fa"} Nov 03 16:14:26.664589 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:26.664555 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" event={"ID":"2c812633-b15d-4a30-982d-80d276ec78c7","Type":"ContainerStarted","Data":"e0267110fa5785583c5bc27af80c5d6b1118f900db5d3576a7f80d4b17d8ac40"} Nov 03 16:14:27.670591 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:27.670556 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xfk72" event={"ID":"5ced7a93-d9f0-44de-9a76-34a84e41843b","Type":"ContainerStarted","Data":"d110310e15893394c7b883af31f8db5e7d27be17daef26394cea90d7eb6e0f1c"} Nov 03 16:14:27.671020 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:27.670636 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xfk72" event={"ID":"5ced7a93-d9f0-44de-9a76-34a84e41843b","Type":"ContainerStarted","Data":"95db0e14929d5f48730dd9a960686669ef4fa32af7cd3101965ba7dee153c3b2"} Nov 03 16:14:27.671020 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:27.670683 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:27.673255 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:27.673224 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-r755z" event={"ID":"2c812633-b15d-4a30-982d-80d276ec78c7","Type":"ContainerStarted","Data":"8920d5fce33fe721fa934f7cad650a95e1255937e3cfa6754f2ce26382b4d325"} Nov 03 16:14:29.558297 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:29.558269 4947 scope.go:122] "RemoveContainer" containerID="b28d6b7ec12d483811bb785fad726a2c5a2c327c8c52ad567df729285fe95977" Nov 03 16:14:29.685639 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:29.685605 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:14:29.685836 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:29.685731 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" event={"ID":"216c0d66-a3ec-4748-b761-3420536bfa3a","Type":"ContainerStarted","Data":"95eb785820d71c562d3c836410c67ad40ed9fa5d4bb5d729415a289b30a828f4"} Nov 03 16:14:29.686071 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:29.686051 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:29.687050 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:29.687025 4947 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-tnh8w container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused" start-of-body= Nov 03 16:14:29.687143 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:29.687081 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" podUID="216c0d66-a3ec-4748-b761-3420536bfa3a" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused" Nov 03 16:14:30.016816 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:30.016787 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zmzv6" Nov 03 16:14:30.692858 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:30.692828 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tnh8w" Nov 03 16:14:33.447459 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.447421 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:33.447459 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.447459 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:33.447985 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.447516 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:33.447985 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.447557 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:33.447985 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.447603 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:33.450445 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.450411 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f02c434e-ede0-4bea-8830-f3444c21a6a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rj5rd\" (UID: \"f02c434e-ede0-4bea-8830-f3444c21a6a6\") " pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:33.450445 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.450439 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66b43213-9ec2-4164-9f8d-d7621bd57eb3-metrics-certs\") pod \"router-default-5dc56589f5-wsj6w\" (UID: \"66b43213-9ec2-4164-9f8d-d7621bd57eb3\") " pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:33.450625 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.450473 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls\") pod \"image-registry-68c8fc97c8-vxzz9\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:33.450759 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.450740 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-bw6bt\" (UID: \"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:33.450800 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.450783 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls\") pod \"image-registry-7d945bd975-wtl6x\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:33.459254 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.459224 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" Nov 03 16:14:33.460320 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.460300 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:33.461323 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.461304 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:33.466989 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.466968 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:33.470016 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.469999 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rj5rd" Nov 03 16:14:33.812316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.812291 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt"] Nov 03 16:14:33.815202 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:33.815171 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3af86a0_c2cd_4cbb_8143_49e9ae2e9b28.slice/crio-718559a5f85a643ff7be7ade76ffd04e03836f34fd5f03993b47dbe52d73e575 WatchSource:0}: Error finding container 718559a5f85a643ff7be7ade76ffd04e03836f34fd5f03993b47dbe52d73e575: Status 404 returned error can't find the container with id 718559a5f85a643ff7be7ade76ffd04e03836f34fd5f03993b47dbe52d73e575 Nov 03 16:14:33.825908 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.825882 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5dc56589f5-wsj6w"] Nov 03 16:14:33.826401 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:33.826376 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b43213_9ec2_4164_9f8d_d7621bd57eb3.slice/crio-87390aa33d002aff3260289c2b3a6db18af71c0c0dca0b46e368bc70011cb48c WatchSource:0}: Error finding container 87390aa33d002aff3260289c2b3a6db18af71c0c0dca0b46e368bc70011cb48c: Status 404 returned error can't find the container with id 87390aa33d002aff3260289c2b3a6db18af71c0c0dca0b46e368bc70011cb48c Nov 03 16:14:33.830406 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.830380 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d945bd975-wtl6x"] Nov 03 16:14:33.835866 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:33.835817 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03cc593d_32bc_4453_8a45_2b546f862acf.slice/crio-f8d6fe706180bf38d1b704e247c280d8948793c0ad913520cb23488ba6669876 WatchSource:0}: Error finding container f8d6fe706180bf38d1b704e247c280d8948793c0ad913520cb23488ba6669876: Status 404 returned error can't find the container with id f8d6fe706180bf38d1b704e247c280d8948793c0ad913520cb23488ba6669876 Nov 03 16:14:33.851935 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.851893 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rj5rd"] Nov 03 16:14:33.860284 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:33.856823 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68c8fc97c8-vxzz9"] Nov 03 16:14:33.860284 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:33.859171 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13e4eb2_b4f2_4f66_85d3_8bedb078823c.slice/crio-4fdd43cb92fc0a9b3f9367fa86bad4f9f0b4e609c6fa0de46218901a848a5487 WatchSource:0}: Error finding container 4fdd43cb92fc0a9b3f9367fa86bad4f9f0b4e609c6fa0de46218901a848a5487: Status 404 returned error can't find the container with id 4fdd43cb92fc0a9b3f9367fa86bad4f9f0b4e609c6fa0de46218901a848a5487 Nov 03 16:14:34.706779 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:34.706680 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d945bd975-wtl6x"] Nov 03 16:14:34.710227 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:34.710187 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rj5rd" event={"ID":"f02c434e-ede0-4bea-8830-f3444c21a6a6","Type":"ContainerStarted","Data":"7f0fc937c70e95418c6e07f2f7f6cfaa748a8130f7b6dfc17186a438cca4e214"} Nov 03 16:14:34.710379 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:34.710266 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rj5rd" event={"ID":"f02c434e-ede0-4bea-8830-f3444c21a6a6","Type":"ContainerStarted","Data":"518615c6c8fe3f36c565fa864ce6841952ebc144fcd240168750848a48c28b4c"} Nov 03 16:14:34.710379 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:34.710296 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rj5rd" event={"ID":"f02c434e-ede0-4bea-8830-f3444c21a6a6","Type":"ContainerStarted","Data":"e965b53bc1ca3443cd8bc55e82666d8d96fb05c1287d197a390ca1a71d801d46"} Nov 03 16:14:34.712403 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:34.712364 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5dc56589f5-wsj6w" event={"ID":"66b43213-9ec2-4164-9f8d-d7621bd57eb3","Type":"ContainerStarted","Data":"9e9b75dd6f46288886dab2198903169feb924879fe1d05a66005e78d232a14b2"} Nov 03 16:14:34.712623 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:34.712412 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5dc56589f5-wsj6w" event={"ID":"66b43213-9ec2-4164-9f8d-d7621bd57eb3","Type":"ContainerStarted","Data":"87390aa33d002aff3260289c2b3a6db18af71c0c0dca0b46e368bc70011cb48c"} Nov 03 16:14:34.714321 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:34.714245 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" event={"ID":"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28","Type":"ContainerStarted","Data":"718559a5f85a643ff7be7ade76ffd04e03836f34fd5f03993b47dbe52d73e575"} Nov 03 16:14:34.717001 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:34.716965 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" event={"ID":"a13e4eb2-b4f2-4f66-85d3-8bedb078823c","Type":"ContainerStarted","Data":"cfe1943f799b64d274e9b2f216785d559e2b9d328253c0e33e4cb28a8ee6c3f6"} Nov 03 16:14:34.717149 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:34.717019 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" event={"ID":"a13e4eb2-b4f2-4f66-85d3-8bedb078823c","Type":"ContainerStarted","Data":"4fdd43cb92fc0a9b3f9367fa86bad4f9f0b4e609c6fa0de46218901a848a5487"} Nov 03 16:14:34.717490 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:34.717474 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:34.719216 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:34.719195 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" event={"ID":"03cc593d-32bc-4453-8a45-2b546f862acf","Type":"ContainerStarted","Data":"a0429bef9df275333712411248fc51d07880fd6f84c5c09316b50f16be261f45"} Nov 03 16:14:34.719310 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:34.719230 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" event={"ID":"03cc593d-32bc-4453-8a45-2b546f862acf","Type":"ContainerStarted","Data":"f8d6fe706180bf38d1b704e247c280d8948793c0ad913520cb23488ba6669876"} Nov 03 16:14:34.719362 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:34.719333 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:35.460867 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:35.460831 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:35.464248 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:35.464219 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:35.726512 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:35.726419 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-bw6bt" event={"ID":"e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28","Type":"ContainerStarted","Data":"9fcfc6cc6742594e8552029c29e8fc79b648cdda485c3a15f8f9c12ae8a03c9f"} Nov 03 16:14:35.726993 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:35.726719 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:35.728030 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:35.728002 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5dc56589f5-wsj6w" Nov 03 16:14:36.732632 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:36.732593 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rj5rd" event={"ID":"f02c434e-ede0-4bea-8830-f3444c21a6a6","Type":"ContainerStarted","Data":"615f720b3484a76d4508c201545c1e35b924670b316b1797096f37ba603000b4"} Nov 03 16:14:37.678588 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:37.678560 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xfk72" Nov 03 16:14:44.179881 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.179847 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gvrm4" Nov 03 16:14:44.514208 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.514167 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k8vg5"] Nov 03 16:14:44.550505 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.550451 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.574687 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.574659 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 16:14:44.575113 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.575096 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cpmbj\"" Nov 03 16:14:44.575490 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.575465 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 16:14:44.575755 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.575738 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 16:14:44.575832 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.575794 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 16:14:44.651665 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.651615 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-sys\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.651665 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.651661 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-node-exporter-tls\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.651968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.651689 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-root\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.651968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.651816 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.651968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.651850 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-node-exporter-wtmp\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.651968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.651894 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-node-exporter-textfile\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.651968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.651913 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-metrics-client-ca\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.652198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.652071 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-node-exporter-accelerators-collector-config\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.652198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.652121 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl5q7\" (UniqueName: \"kubernetes.io/projected/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-kube-api-access-rl5q7\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.753316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.753266 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-node-exporter-wtmp\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.753506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.753337 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-node-exporter-textfile\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.753506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.753355 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-metrics-client-ca\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.753506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.753379 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-node-exporter-accelerators-collector-config\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.753506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.753407 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl5q7\" (UniqueName: \"kubernetes.io/projected/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-kube-api-access-rl5q7\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.753506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.753435 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-sys\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.753506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.753445 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-node-exporter-wtmp\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.753506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.753462 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-node-exporter-tls\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.753887 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.753514 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-root\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.753887 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.753518 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-sys\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.753887 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.753563 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.753887 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.753632 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-root\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.753887 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.753806 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-node-exporter-textfile\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.754101 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.754001 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-node-exporter-accelerators-collector-config\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.754101 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.754064 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-metrics-client-ca\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.756096 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.756072 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.756182 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.756141 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-node-exporter-tls\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.777834 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.777775 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl5q7\" (UniqueName: \"kubernetes.io/projected/adfb8fe7-01d3-4c9c-a933-df17b1602ca6-kube-api-access-rl5q7\") pod \"node-exporter-k8vg5\" (UID: \"adfb8fe7-01d3-4c9c-a933-df17b1602ca6\") " pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.902840 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:44.902811 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k8vg5" Nov 03 16:14:44.945219 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:14:44.945180 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfb8fe7_01d3_4c9c_a933_df17b1602ca6.slice/crio-e37246edd0923e34f2e7c48edefcf6b0430519c5f101f3fea649949926057b6c WatchSource:0}: Error finding container e37246edd0923e34f2e7c48edefcf6b0430519c5f101f3fea649949926057b6c: Status 404 returned error can't find the container with id e37246edd0923e34f2e7c48edefcf6b0430519c5f101f3fea649949926057b6c Nov 03 16:14:45.771152 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:45.771092 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8vg5" event={"ID":"adfb8fe7-01d3-4c9c-a933-df17b1602ca6","Type":"ContainerStarted","Data":"e37246edd0923e34f2e7c48edefcf6b0430519c5f101f3fea649949926057b6c"} Nov 03 16:14:46.539800 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:46.539767 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5fq8q" Nov 03 16:14:46.777545 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:46.777505 4947 generic.go:356] "Generic (PLEG): container finished" podID="adfb8fe7-01d3-4c9c-a933-df17b1602ca6" containerID="991373c7648148826b27796b4cc36014a7de7745185ab7a41329ec3d7be04d00" exitCode=0 Nov 03 16:14:46.778070 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:46.777587 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8vg5" event={"ID":"adfb8fe7-01d3-4c9c-a933-df17b1602ca6","Type":"ContainerDied","Data":"991373c7648148826b27796b4cc36014a7de7745185ab7a41329ec3d7be04d00"} Nov 03 16:14:47.783804 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:47.783765 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8vg5" event={"ID":"adfb8fe7-01d3-4c9c-a933-df17b1602ca6","Type":"ContainerStarted","Data":"a03492c27cc38e92e73a926489107f1e02eb4423a49e60f1689949d6430a10a8"} Nov 03 16:14:47.784293 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:47.783868 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k8vg5" event={"ID":"adfb8fe7-01d3-4c9c-a933-df17b1602ca6","Type":"ContainerStarted","Data":"982b28a7f6cc788535d1a6484f279b97385b4006fa0368bf8806f6082b5beb41"} Nov 03 16:14:47.874243 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:47.874190 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k8vg5" podStartSLOduration=2.5290369520000002 podStartE2EDuration="3.8741764s" totalImagesPullingTime="1.258463078s" totalInitContainerRuntime="86.67637ms" isStatefulPod=true podCreationTimestamp="2026-11-03 16:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:14:47.873396602 +0000 UTC m=+31.462396570" watchObservedRunningTime="2026-11-03 16:14:47.8741764 +0000 UTC m=+31.463176363" Nov 03 16:14:55.732557 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:55.732527 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:14:56.736655 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:56.736629 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:14:57.708576 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:14:57.708491 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68c8fc97c8-vxzz9"] Nov 03 16:15:00.129989 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:15:00.129941 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:15:00.750776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:00.750739 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" podUID="03cc593d-32bc-4453-8a45-2b546f862acf" containerName="registry" containerID="cri-o://a0429bef9df275333712411248fc51d07880fd6f84c5c09316b50f16be261f45" gracePeriod=30 Nov 03 16:15:01.000076 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.000055 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:15:01.095271 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.095196 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-installation-pull-secrets\") pod \"03cc593d-32bc-4453-8a45-2b546f862acf\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " Nov 03 16:15:01.095271 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.095251 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-registry-certificates\") pod \"03cc593d-32bc-4453-8a45-2b546f862acf\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " Nov 03 16:15:01.095507 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.095273 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls\") pod \"03cc593d-32bc-4453-8a45-2b546f862acf\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " Nov 03 16:15:01.095507 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.095303 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-kube-api-access-rl4vw\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-kube-api-access-rl4vw\") pod \"03cc593d-32bc-4453-8a45-2b546f862acf\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " Nov 03 16:15:01.095507 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.095341 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03cc593d-32bc-4453-8a45-2b546f862acf-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03cc593d-32bc-4453-8a45-2b546f862acf-ca-trust-extracted\") pod \"03cc593d-32bc-4453-8a45-2b546f862acf\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " Nov 03 16:15:01.095507 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.095381 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-image-registry-private-configuration\") pod \"03cc593d-32bc-4453-8a45-2b546f862acf\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " Nov 03 16:15:01.095507 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.095400 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-trusted-ca\") pod \"03cc593d-32bc-4453-8a45-2b546f862acf\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " Nov 03 16:15:01.095507 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.095424 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-bound-sa-token\") pod \"03cc593d-32bc-4453-8a45-2b546f862acf\" (UID: \"03cc593d-32bc-4453-8a45-2b546f862acf\") " Nov 03 16:15:01.095905 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.095845 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-registry-certificates" pod "03cc593d-32bc-4453-8a45-2b546f862acf" (UID: "03cc593d-32bc-4453-8a45-2b546f862acf"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:01.096214 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.096175 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-trusted-ca" pod "03cc593d-32bc-4453-8a45-2b546f862acf" (UID: "03cc593d-32bc-4453-8a45-2b546f862acf"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:01.098602 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.098574 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-installation-pull-secrets" pod "03cc593d-32bc-4453-8a45-2b546f862acf" (UID: "03cc593d-32bc-4453-8a45-2b546f862acf"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:01.098602 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.098585 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-kube-api-access-rl4vw" pod "03cc593d-32bc-4453-8a45-2b546f862acf" (UID: "03cc593d-32bc-4453-8a45-2b546f862acf"). InnerVolumeSpecName "kube-api-access-rl4vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:15:01.098929 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.098674 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls" pod "03cc593d-32bc-4453-8a45-2b546f862acf" (UID: "03cc593d-32bc-4453-8a45-2b546f862acf"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:15:01.098929 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.098678 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-bound-sa-token" pod "03cc593d-32bc-4453-8a45-2b546f862acf" (UID: "03cc593d-32bc-4453-8a45-2b546f862acf"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:15:01.098929 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.098685 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-image-registry-private-configuration" pod "03cc593d-32bc-4453-8a45-2b546f862acf" (UID: "03cc593d-32bc-4453-8a45-2b546f862acf"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:01.103018 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.102899 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03cc593d-32bc-4453-8a45-2b546f862acf-ca-trust-extracted" pod "03cc593d-32bc-4453-8a45-2b546f862acf" (UID: "03cc593d-32bc-4453-8a45-2b546f862acf"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:01.196131 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.196092 4947 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-trusted-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:15:01.196131 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.196126 4947 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-bound-sa-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:15:01.196131 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.196137 4947 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-installation-pull-secrets\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:15:01.196604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.196150 4947 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/03cc593d-32bc-4453-8a45-2b546f862acf-registry-certificates\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:15:01.196604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.196160 4947 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-registry-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:15:01.196604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.196169 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl4vw\" (UniqueName: \"kubernetes.io/projected/03cc593d-32bc-4453-8a45-2b546f862acf-kube-api-access-rl4vw\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:15:01.196604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.196178 4947 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/03cc593d-32bc-4453-8a45-2b546f862acf-ca-trust-extracted\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:15:01.196604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.196189 4947 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/03cc593d-32bc-4453-8a45-2b546f862acf-image-registry-private-configuration\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:15:01.842133 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.842095 4947 generic.go:356] "Generic (PLEG): container finished" podID="03cc593d-32bc-4453-8a45-2b546f862acf" containerID="a0429bef9df275333712411248fc51d07880fd6f84c5c09316b50f16be261f45" exitCode=0 Nov 03 16:15:01.842304 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.842201 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" event={"ID":"03cc593d-32bc-4453-8a45-2b546f862acf","Type":"ContainerDied","Data":"a0429bef9df275333712411248fc51d07880fd6f84c5c09316b50f16be261f45"} Nov 03 16:15:01.842304 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.842230 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" Nov 03 16:15:01.842304 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.842291 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d945bd975-wtl6x" event={"ID":"03cc593d-32bc-4453-8a45-2b546f862acf","Type":"ContainerDied","Data":"f8d6fe706180bf38d1b704e247c280d8948793c0ad913520cb23488ba6669876"} Nov 03 16:15:01.842409 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.842320 4947 scope.go:122] "RemoveContainer" containerID="a0429bef9df275333712411248fc51d07880fd6f84c5c09316b50f16be261f45" Nov 03 16:15:01.851075 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.851058 4947 scope.go:122] "RemoveContainer" containerID="a0429bef9df275333712411248fc51d07880fd6f84c5c09316b50f16be261f45" Nov 03 16:15:01.851366 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:15:01.851346 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0429bef9df275333712411248fc51d07880fd6f84c5c09316b50f16be261f45\": container with ID starting with a0429bef9df275333712411248fc51d07880fd6f84c5c09316b50f16be261f45 not found: ID does not exist" containerID="a0429bef9df275333712411248fc51d07880fd6f84c5c09316b50f16be261f45" Nov 03 16:15:01.851404 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.851376 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0429bef9df275333712411248fc51d07880fd6f84c5c09316b50f16be261f45"} err="failed to get container status \"a0429bef9df275333712411248fc51d07880fd6f84c5c09316b50f16be261f45\": rpc error: code = NotFound desc = could not find container \"a0429bef9df275333712411248fc51d07880fd6f84c5c09316b50f16be261f45\": container with ID starting with a0429bef9df275333712411248fc51d07880fd6f84c5c09316b50f16be261f45 not found: ID does not exist" Nov 03 16:15:01.910590 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.910556 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7d945bd975-wtl6x"] Nov 03 16:15:01.924077 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:01.924044 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7d945bd975-wtl6x"] Nov 03 16:15:02.562969 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:02.562940 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03cc593d-32bc-4453-8a45-2b546f862acf" path="/var/lib/kubelet/pods/03cc593d-32bc-4453-8a45-2b546f862acf/volumes" Nov 03 16:15:21.919736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:21.919680 4947 generic.go:356] "Generic (PLEG): container finished" podID="b9c106a2-ce74-4c57-80c1-b1f75da0d43d" containerID="087377bd323389ac129c34915b45f033a9599717a55cf079aa2b973df0b1983c" exitCode=0 Nov 03 16:15:21.920416 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:21.919772 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-964q9" event={"ID":"b9c106a2-ce74-4c57-80c1-b1f75da0d43d","Type":"ContainerDied","Data":"087377bd323389ac129c34915b45f033a9599717a55cf079aa2b973df0b1983c"} Nov 03 16:15:21.920416 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:21.920173 4947 scope.go:122] "RemoveContainer" containerID="087377bd323389ac129c34915b45f033a9599717a55cf079aa2b973df0b1983c" Nov 03 16:15:22.729289 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:22.729249 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" podUID="a13e4eb2-b4f2-4f66-85d3-8bedb078823c" containerName="registry" containerID="cri-o://cfe1943f799b64d274e9b2f216785d559e2b9d328253c0e33e4cb28a8ee6c3f6" gracePeriod=30 Nov 03 16:15:22.925860 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:22.925826 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-964q9" event={"ID":"b9c106a2-ce74-4c57-80c1-b1f75da0d43d","Type":"ContainerStarted","Data":"85e35fa4118ca6b231501fb111ac353af54e1d4a430dc12e04e4f3c5a34b0b28"} Nov 03 16:15:22.927768 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:22.927734 4947 generic.go:356] "Generic (PLEG): container finished" podID="a13e4eb2-b4f2-4f66-85d3-8bedb078823c" containerID="cfe1943f799b64d274e9b2f216785d559e2b9d328253c0e33e4cb28a8ee6c3f6" exitCode=0 Nov 03 16:15:22.927902 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:22.927782 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" event={"ID":"a13e4eb2-b4f2-4f66-85d3-8bedb078823c","Type":"ContainerDied","Data":"cfe1943f799b64d274e9b2f216785d559e2b9d328253c0e33e4cb28a8ee6c3f6"} Nov 03 16:15:22.964656 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:22.964631 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:15:23.070163 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.070129 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-installation-pull-secrets\") pod \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " Nov 03 16:15:23.070163 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.070164 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-kube-api-access-z4dbx\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-kube-api-access-z4dbx\") pod \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " Nov 03 16:15:23.070388 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.070192 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-ca-trust-extracted\") pod \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " Nov 03 16:15:23.070388 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.070237 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls\") pod \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " Nov 03 16:15:23.070388 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.070264 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-trusted-ca\") pod \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " Nov 03 16:15:23.070388 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.070289 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-image-registry-private-configuration\") pod \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " Nov 03 16:15:23.070388 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.070353 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-certificates\") pod \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " Nov 03 16:15:23.070388 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.070383 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-bound-sa-token\") pod \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\" (UID: \"a13e4eb2-b4f2-4f66-85d3-8bedb078823c\") " Nov 03 16:15:23.070945 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.070867 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-certificates" pod "a13e4eb2-b4f2-4f66-85d3-8bedb078823c" (UID: "a13e4eb2-b4f2-4f66-85d3-8bedb078823c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:23.070945 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.070898 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-trusted-ca" pod "a13e4eb2-b4f2-4f66-85d3-8bedb078823c" (UID: "a13e4eb2-b4f2-4f66-85d3-8bedb078823c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:15:23.073030 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.073001 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-bound-sa-token" pod "a13e4eb2-b4f2-4f66-85d3-8bedb078823c" (UID: "a13e4eb2-b4f2-4f66-85d3-8bedb078823c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:15:23.073167 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.073001 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls" pod "a13e4eb2-b4f2-4f66-85d3-8bedb078823c" (UID: "a13e4eb2-b4f2-4f66-85d3-8bedb078823c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:15:23.073167 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.073100 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-installation-pull-secrets" pod "a13e4eb2-b4f2-4f66-85d3-8bedb078823c" (UID: "a13e4eb2-b4f2-4f66-85d3-8bedb078823c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:23.073425 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.073404 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-kube-api-access-z4dbx" pod "a13e4eb2-b4f2-4f66-85d3-8bedb078823c" (UID: "a13e4eb2-b4f2-4f66-85d3-8bedb078823c"). InnerVolumeSpecName "kube-api-access-z4dbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:15:23.073425 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.073413 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-image-registry-private-configuration" pod "a13e4eb2-b4f2-4f66-85d3-8bedb078823c" (UID: "a13e4eb2-b4f2-4f66-85d3-8bedb078823c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:15:23.077358 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.077338 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-ca-trust-extracted" pod "a13e4eb2-b4f2-4f66-85d3-8bedb078823c" (UID: "a13e4eb2-b4f2-4f66-85d3-8bedb078823c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:15:23.171638 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.171600 4947 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-bound-sa-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:15:23.171638 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.171634 4947 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-installation-pull-secrets\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:15:23.171638 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.171646 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4dbx\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-kube-api-access-z4dbx\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:15:23.171913 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.171659 4947 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-ca-trust-extracted\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:15:23.171913 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.171672 4947 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:15:23.171913 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.171685 4947 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-trusted-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:15:23.171913 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.171714 4947 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-image-registry-private-configuration\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:15:23.171913 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.171727 4947 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a13e4eb2-b4f2-4f66-85d3-8bedb078823c-registry-certificates\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:15:23.932855 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.932816 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" Nov 03 16:15:23.932855 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.932822 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68c8fc97c8-vxzz9" event={"ID":"a13e4eb2-b4f2-4f66-85d3-8bedb078823c","Type":"ContainerDied","Data":"4fdd43cb92fc0a9b3f9367fa86bad4f9f0b4e609c6fa0de46218901a848a5487"} Nov 03 16:15:23.933407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:23.932895 4947 scope.go:122] "RemoveContainer" containerID="cfe1943f799b64d274e9b2f216785d559e2b9d328253c0e33e4cb28a8ee6c3f6" Nov 03 16:15:24.002935 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:24.002903 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68c8fc97c8-vxzz9"] Nov 03 16:15:24.016475 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:24.016436 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68c8fc97c8-vxzz9"] Nov 03 16:15:24.562713 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:24.562672 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a13e4eb2-b4f2-4f66-85d3-8bedb078823c" path="/var/lib/kubelet/pods/a13e4eb2-b4f2-4f66-85d3-8bedb078823c/volumes" Nov 03 16:15:30.131095 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:15:30.131053 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:15:36.982611 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:36.982573 4947 generic.go:356] "Generic (PLEG): container finished" podID="bf8c316f-51ce-4c69-865f-1d481ff30630" containerID="622c06122a7b144c63fe80289c612a76c433e21a9d36aeab2e9ac514329c1fb4" exitCode=0 Nov 03 16:15:36.983058 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:36.982652 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" event={"ID":"bf8c316f-51ce-4c69-865f-1d481ff30630","Type":"ContainerDied","Data":"622c06122a7b144c63fe80289c612a76c433e21a9d36aeab2e9ac514329c1fb4"} Nov 03 16:15:36.983058 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:36.982985 4947 scope.go:122] "RemoveContainer" containerID="622c06122a7b144c63fe80289c612a76c433e21a9d36aeab2e9ac514329c1fb4" Nov 03 16:15:37.988114 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:37.988078 4947 generic.go:356] "Generic (PLEG): container finished" podID="1580792e-eb86-4d79-92a7-8055e6a8aaff" containerID="30a9ebb9c659c99b128358965f83114a59d103a42daf52c9a50fa1427b14e76f" exitCode=0 Nov 03 16:15:37.988555 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:37.988151 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" event={"ID":"1580792e-eb86-4d79-92a7-8055e6a8aaff","Type":"ContainerDied","Data":"30a9ebb9c659c99b128358965f83114a59d103a42daf52c9a50fa1427b14e76f"} Nov 03 16:15:37.988555 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:37.988510 4947 scope.go:122] "RemoveContainer" containerID="30a9ebb9c659c99b128358965f83114a59d103a42daf52c9a50fa1427b14e76f" Nov 03 16:15:37.990226 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:37.990204 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-zn6z4" event={"ID":"bf8c316f-51ce-4c69-865f-1d481ff30630","Type":"ContainerStarted","Data":"09077a15cda58c039c91ad907ced7007a5952594177872c08d554ba065e644a7"} Nov 03 16:15:38.996389 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:15:38.996357 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-qqcft" event={"ID":"1580792e-eb86-4d79-92a7-8055e6a8aaff","Type":"ContainerStarted","Data":"f17c0c0e272a79d8782c816cab314520439fcbd383a04392b994dbed79cb1dca"} Nov 03 16:16:00.129684 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:16:00.129647 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:16:30.130663 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:16:30.130581 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:17:00.128999 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:17:00.128963 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:17:30.129155 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:17:30.129115 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:18:00.128985 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:18:00.128901 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:18:30.129646 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:18:30.129604 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:18:30.858613 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.858577 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd"] Nov 03 16:18:30.858882 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.858869 4947 cpu_manager.go:414] "containerMap: removing container" podUID="03cc593d-32bc-4453-8a45-2b546f862acf" containerName="registry" Nov 03 16:18:30.858932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.858884 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03cc593d-32bc-4453-8a45-2b546f862acf" containerName="registry" Nov 03 16:18:30.858932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.858903 4947 cpu_manager.go:414] "containerMap: removing container" podUID="a13e4eb2-b4f2-4f66-85d3-8bedb078823c" containerName="registry" Nov 03 16:18:30.858932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.858908 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13e4eb2-b4f2-4f66-85d3-8bedb078823c" containerName="registry" Nov 03 16:18:30.859025 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.858969 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="03cc593d-32bc-4453-8a45-2b546f862acf" containerName="registry" Nov 03 16:18:30.859025 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.858978 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13e4eb2-b4f2-4f66-85d3-8bedb078823c" containerName="registry" Nov 03 16:18:30.860780 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.860765 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:30.879707 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.879658 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 16:18:30.879881 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.879747 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 16:18:30.879881 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.879864 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 16:18:30.892439 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.892408 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 16:18:30.892626 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.892464 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xzr7j\"" Nov 03 16:18:30.893825 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.893802 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd"] Nov 03 16:18:30.901804 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.901778 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:30.901885 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.901818 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:30.901885 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.901840 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:30.902027 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.901999 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7xmm\" (UniqueName: \"kubernetes.io/projected/938147b7-b53f-439b-a261-7d0d81dcf7bb-kube-api-access-j7xmm\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:30.902081 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.902064 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:30.902127 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.902114 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/938147b7-b53f-439b-a261-7d0d81dcf7bb-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:30.902189 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:30.902170 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:31.003205 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:31.003162 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:31.003205 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:31.003210 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7xmm\" (UniqueName: \"kubernetes.io/projected/938147b7-b53f-439b-a261-7d0d81dcf7bb-kube-api-access-j7xmm\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:31.003486 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:31.003241 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:31.003486 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:31.003267 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/938147b7-b53f-439b-a261-7d0d81dcf7bb-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:31.003486 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:31.003282 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:31.003640 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:31.003523 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:31.003640 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:31.003578 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:31.004112 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:31.004083 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:31.005866 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:31.005846 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/938147b7-b53f-439b-a261-7d0d81dcf7bb-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:31.006012 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:31.005991 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:31.006316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:31.006297 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:31.006430 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:31.006410 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:31.029539 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:31.029503 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7xmm\" (UniqueName: \"kubernetes.io/projected/938147b7-b53f-439b-a261-7d0d81dcf7bb-kube-api-access-j7xmm\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:31.032542 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:31.032516 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-xclsd\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:31.171190 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:31.171091 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:31.497338 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:31.497314 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd"] Nov 03 16:18:31.499518 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:18:31.499494 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938147b7_b53f_439b_a261_7d0d81dcf7bb.slice/crio-9085202ae8e522924560eec2d49d21a13c426906462d35017c6dab17427ec309 WatchSource:0}: Error finding container 9085202ae8e522924560eec2d49d21a13c426906462d35017c6dab17427ec309: Status 404 returned error can't find the container with id 9085202ae8e522924560eec2d49d21a13c426906462d35017c6dab17427ec309 Nov 03 16:18:31.622282 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:31.622245 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" event={"ID":"938147b7-b53f-439b-a261-7d0d81dcf7bb","Type":"ContainerStarted","Data":"9085202ae8e522924560eec2d49d21a13c426906462d35017c6dab17427ec309"} Nov 03 16:18:34.192249 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:34.192206 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:18:34.192552 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:34.192283 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:18:34.636768 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:34.636722 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" event={"ID":"938147b7-b53f-439b-a261-7d0d81dcf7bb","Type":"ContainerStarted","Data":"152b82e2ab98c4bbc3963c895aeb892ffb57d05d9e8019b5d383358e925fd866"} Nov 03 16:18:34.636959 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:34.636842 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:18:34.714268 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:34.714196 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" podStartSLOduration=2.024243564 podStartE2EDuration="4.714176073s" totalImagesPullingTime="2.689932509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:18:34.700227953 +0000 UTC m=+258.289227922" watchObservedRunningTime="2026-11-03 16:18:34.714176073 +0000 UTC m=+258.303176046" Nov 03 16:18:35.644785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:35.644743 4947 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-xclsd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:18:35.645175 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:35.644838 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" podUID="938147b7-b53f-439b-a261-7d0d81dcf7bb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:18:38.644410 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:18:38.644377 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:19:00.129448 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:19:00.129409 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:19:00.296258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.296216 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr"] Nov 03 16:19:00.299857 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.299828 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.340116 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.340076 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr"] Nov 03 16:19:00.359416 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.359385 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 16:19:00.359602 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.359434 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 16:19:00.359602 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.359482 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 16:19:00.359678 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.359659 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-k6tn8\"" Nov 03 16:19:00.384256 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.384010 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 16:19:00.384256 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.384012 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 16:19:00.434914 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.434875 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c8fe26c-c295-400c-84a5-857b478674cd-tmp\") pod \"opendatahub-operator-controller-manager-698cf769bf-l8kvr\" (UID: \"1c8fe26c-c295-400c-84a5-857b478674cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.435117 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.434931 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c8fe26c-c295-400c-84a5-857b478674cd-metrics-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-l8kvr\" (UID: \"1c8fe26c-c295-400c-84a5-857b478674cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.435117 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.435038 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c8fe26c-c295-400c-84a5-857b478674cd-manifests\") pod \"opendatahub-operator-controller-manager-698cf769bf-l8kvr\" (UID: \"1c8fe26c-c295-400c-84a5-857b478674cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.435117 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.435067 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9wnh\" (UniqueName: \"kubernetes.io/projected/1c8fe26c-c295-400c-84a5-857b478674cd-kube-api-access-w9wnh\") pod \"opendatahub-operator-controller-manager-698cf769bf-l8kvr\" (UID: \"1c8fe26c-c295-400c-84a5-857b478674cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.435117 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.435099 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c8fe26c-c295-400c-84a5-857b478674cd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-l8kvr\" (UID: \"1c8fe26c-c295-400c-84a5-857b478674cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.435283 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.435196 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c8fe26c-c295-400c-84a5-857b478674cd-webhook-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-l8kvr\" (UID: \"1c8fe26c-c295-400c-84a5-857b478674cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.535577 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.535543 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c8fe26c-c295-400c-84a5-857b478674cd-metrics-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-l8kvr\" (UID: \"1c8fe26c-c295-400c-84a5-857b478674cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.535788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.535585 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c8fe26c-c295-400c-84a5-857b478674cd-manifests\") pod \"opendatahub-operator-controller-manager-698cf769bf-l8kvr\" (UID: \"1c8fe26c-c295-400c-84a5-857b478674cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.535788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.535609 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9wnh\" (UniqueName: \"kubernetes.io/projected/1c8fe26c-c295-400c-84a5-857b478674cd-kube-api-access-w9wnh\") pod \"opendatahub-operator-controller-manager-698cf769bf-l8kvr\" (UID: \"1c8fe26c-c295-400c-84a5-857b478674cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.535788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.535650 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c8fe26c-c295-400c-84a5-857b478674cd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-l8kvr\" (UID: \"1c8fe26c-c295-400c-84a5-857b478674cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.535788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.535686 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c8fe26c-c295-400c-84a5-857b478674cd-webhook-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-l8kvr\" (UID: \"1c8fe26c-c295-400c-84a5-857b478674cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.535788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.535752 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c8fe26c-c295-400c-84a5-857b478674cd-tmp\") pod \"opendatahub-operator-controller-manager-698cf769bf-l8kvr\" (UID: \"1c8fe26c-c295-400c-84a5-857b478674cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.536091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.536062 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1c8fe26c-c295-400c-84a5-857b478674cd-manifests\") pod \"opendatahub-operator-controller-manager-698cf769bf-l8kvr\" (UID: \"1c8fe26c-c295-400c-84a5-857b478674cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.538214 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.538189 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c8fe26c-c295-400c-84a5-857b478674cd-tmp\") pod \"opendatahub-operator-controller-manager-698cf769bf-l8kvr\" (UID: \"1c8fe26c-c295-400c-84a5-857b478674cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.538430 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.538402 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1c8fe26c-c295-400c-84a5-857b478674cd-webhook-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-l8kvr\" (UID: \"1c8fe26c-c295-400c-84a5-857b478674cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.538670 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.538650 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1c8fe26c-c295-400c-84a5-857b478674cd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-l8kvr\" (UID: \"1c8fe26c-c295-400c-84a5-857b478674cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.538779 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.538763 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/1c8fe26c-c295-400c-84a5-857b478674cd-metrics-cert\") pod \"opendatahub-operator-controller-manager-698cf769bf-l8kvr\" (UID: \"1c8fe26c-c295-400c-84a5-857b478674cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.562683 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.562657 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9wnh\" (UniqueName: \"kubernetes.io/projected/1c8fe26c-c295-400c-84a5-857b478674cd-kube-api-access-w9wnh\") pod \"opendatahub-operator-controller-manager-698cf769bf-l8kvr\" (UID: \"1c8fe26c-c295-400c-84a5-857b478674cd\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.614573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.614550 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:00.976843 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:00.976820 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr"] Nov 03 16:19:00.978860 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:19:00.978834 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8fe26c_c295_400c_84a5_857b478674cd.slice/crio-0c4adfc798706f7e7aaecb807327194a1a1924a09a677eeea3a2630ca5c24138 WatchSource:0}: Error finding container 0c4adfc798706f7e7aaecb807327194a1a1924a09a677eeea3a2630ca5c24138: Status 404 returned error can't find the container with id 0c4adfc798706f7e7aaecb807327194a1a1924a09a677eeea3a2630ca5c24138 Nov 03 16:19:01.743429 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:01.743391 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" event={"ID":"1c8fe26c-c295-400c-84a5-857b478674cd","Type":"ContainerStarted","Data":"0c4adfc798706f7e7aaecb807327194a1a1924a09a677eeea3a2630ca5c24138"} Nov 03 16:19:04.761945 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:04.761856 4947 generic.go:356] "Generic (PLEG): container finished" podID="1c8fe26c-c295-400c-84a5-857b478674cd" containerID="0d5556285dd13a2a411541b9e67638a147988aed8be88e9ee3877bb06819697a" exitCode=0 Nov 03 16:19:04.762372 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:04.761954 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" event={"ID":"1c8fe26c-c295-400c-84a5-857b478674cd","Type":"ContainerDied","Data":"0d5556285dd13a2a411541b9e67638a147988aed8be88e9ee3877bb06819697a"} Nov 03 16:19:05.768522 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:05.768490 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" event={"ID":"1c8fe26c-c295-400c-84a5-857b478674cd","Type":"ContainerStarted","Data":"826a03b9544d2db76ecb91d5a3313b157e01920c72735d66b9c3c16f960df057"} Nov 03 16:19:05.768978 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:05.768595 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:05.917865 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:05.917802 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" podStartSLOduration=2.188372716 podStartE2EDuration="5.917784094s" totalImagesPullingTime="3.305654546s" totalInitContainerRuntime="423.756832ms" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:05.90360448 +0000 UTC m=+289.492604449" watchObservedRunningTime="2026-11-03 16:19:05.917784094 +0000 UTC m=+289.506784066" Nov 03 16:19:16.578046 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:16.578010 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:19:16.578489 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:16.578223 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:19:16.774790 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:16.774757 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-698cf769bf-l8kvr" Nov 03 16:19:30.129598 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:19:30.129512 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:19:42.850087 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:42.850050 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-645cdd4567-8ghpl"] Nov 03 16:19:42.856220 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:42.856199 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-645cdd4567-8ghpl" Nov 03 16:19:42.896035 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:42.895999 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-645cdd4567-8ghpl"] Nov 03 16:19:42.929591 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:42.929565 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 16:19:42.932412 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:42.932392 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8rbg9\"" Nov 03 16:19:42.932828 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:42.932815 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 16:19:42.969373 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:42.969344 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa392009-346d-4643-8300-74f3258bc223-tls-certs\") pod \"kube-auth-proxy-645cdd4567-8ghpl\" (UID: \"aa392009-346d-4643-8300-74f3258bc223\") " pod="openshift-ingress/kube-auth-proxy-645cdd4567-8ghpl" Nov 03 16:19:42.969502 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:42.969462 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vf9r\" (UniqueName: \"kubernetes.io/projected/aa392009-346d-4643-8300-74f3258bc223-kube-api-access-6vf9r\") pod \"kube-auth-proxy-645cdd4567-8ghpl\" (UID: \"aa392009-346d-4643-8300-74f3258bc223\") " pod="openshift-ingress/kube-auth-proxy-645cdd4567-8ghpl" Nov 03 16:19:42.969552 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:42.969537 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa392009-346d-4643-8300-74f3258bc223-tmp\") pod \"kube-auth-proxy-645cdd4567-8ghpl\" (UID: \"aa392009-346d-4643-8300-74f3258bc223\") " pod="openshift-ingress/kube-auth-proxy-645cdd4567-8ghpl" Nov 03 16:19:43.070539 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:43.070498 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa392009-346d-4643-8300-74f3258bc223-tmp\") pod \"kube-auth-proxy-645cdd4567-8ghpl\" (UID: \"aa392009-346d-4643-8300-74f3258bc223\") " pod="openshift-ingress/kube-auth-proxy-645cdd4567-8ghpl" Nov 03 16:19:43.070763 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:43.070558 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa392009-346d-4643-8300-74f3258bc223-tls-certs\") pod \"kube-auth-proxy-645cdd4567-8ghpl\" (UID: \"aa392009-346d-4643-8300-74f3258bc223\") " pod="openshift-ingress/kube-auth-proxy-645cdd4567-8ghpl" Nov 03 16:19:43.070763 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:43.070677 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vf9r\" (UniqueName: \"kubernetes.io/projected/aa392009-346d-4643-8300-74f3258bc223-kube-api-access-6vf9r\") pod \"kube-auth-proxy-645cdd4567-8ghpl\" (UID: \"aa392009-346d-4643-8300-74f3258bc223\") " pod="openshift-ingress/kube-auth-proxy-645cdd4567-8ghpl" Nov 03 16:19:43.073028 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:43.073004 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa392009-346d-4643-8300-74f3258bc223-tmp\") pod \"kube-auth-proxy-645cdd4567-8ghpl\" (UID: \"aa392009-346d-4643-8300-74f3258bc223\") " pod="openshift-ingress/kube-auth-proxy-645cdd4567-8ghpl" Nov 03 16:19:43.073310 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:43.073292 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa392009-346d-4643-8300-74f3258bc223-tls-certs\") pod \"kube-auth-proxy-645cdd4567-8ghpl\" (UID: \"aa392009-346d-4643-8300-74f3258bc223\") " pod="openshift-ingress/kube-auth-proxy-645cdd4567-8ghpl" Nov 03 16:19:43.106879 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:43.106808 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vf9r\" (UniqueName: \"kubernetes.io/projected/aa392009-346d-4643-8300-74f3258bc223-kube-api-access-6vf9r\") pod \"kube-auth-proxy-645cdd4567-8ghpl\" (UID: \"aa392009-346d-4643-8300-74f3258bc223\") " pod="openshift-ingress/kube-auth-proxy-645cdd4567-8ghpl" Nov 03 16:19:43.167473 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:43.167426 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-645cdd4567-8ghpl" Nov 03 16:19:43.478166 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:43.478140 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-645cdd4567-8ghpl"] Nov 03 16:19:43.479774 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:19:43.479746 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa392009_346d_4643_8300_74f3258bc223.slice/crio-3c427f8a5c3d355b7a5bc6e5d38ec77be43a27102d87175e631fb092bbed092a WatchSource:0}: Error finding container 3c427f8a5c3d355b7a5bc6e5d38ec77be43a27102d87175e631fb092bbed092a: Status 404 returned error can't find the container with id 3c427f8a5c3d355b7a5bc6e5d38ec77be43a27102d87175e631fb092bbed092a Nov 03 16:19:43.481647 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:43.481631 4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:19:43.916593 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:43.916559 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-645cdd4567-8ghpl" event={"ID":"aa392009-346d-4643-8300-74f3258bc223","Type":"ContainerStarted","Data":"3c427f8a5c3d355b7a5bc6e5d38ec77be43a27102d87175e631fb092bbed092a"} Nov 03 16:19:46.931424 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:46.931388 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-645cdd4567-8ghpl" event={"ID":"aa392009-346d-4643-8300-74f3258bc223","Type":"ContainerStarted","Data":"3388a930976bdd37bac20a352d598c7793d83991f85e11e06f91ff26b51cda4d"} Nov 03 16:19:46.990454 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:19:46.990397 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-645cdd4567-8ghpl" podStartSLOduration=2.559003577 podStartE2EDuration="4.990377938s" totalImagesPullingTime="2.431374361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:19:46.989424701 +0000 UTC m=+330.578424669" watchObservedRunningTime="2026-11-03 16:19:46.990377938 +0000 UTC m=+330.579377913" Nov 03 16:20:00.129845 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:20:00.129802 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:20:30.129378 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:20:30.129339 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:20:43.655660 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.655620 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt"] Nov 03 16:20:43.657982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.657954 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.690015 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.689983 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt"] Nov 03 16:20:43.767295 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.767258 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.767494 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.767300 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.767494 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.767335 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.767494 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.767455 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.767663 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.767498 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfnp8\" (UniqueName: \"kubernetes.io/projected/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-kube-api-access-nfnp8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.767663 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.767598 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.767663 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.767635 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.869025 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.868975 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.869025 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.869034 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.869254 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.869151 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.869254 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.869193 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.869254 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.869237 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.869420 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.869259 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfnp8\" (UniqueName: \"kubernetes.io/projected/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-kube-api-access-nfnp8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.869420 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.869288 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.869888 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.869853 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.871563 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.871538 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.871743 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.871723 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.872000 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.871981 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.872067 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.872013 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.895105 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.895066 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.895244 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.895137 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfnp8\" (UniqueName: \"kubernetes.io/projected/d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316-kube-api-access-nfnp8\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zp4kt\" (UID: \"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:43.968039 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:43.967952 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:44.284012 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:44.283987 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt"] Nov 03 16:20:44.286502 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:20:44.286479 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e6638a_7e4f_4ea6_aa89_2d6a0ec49316.slice/crio-0791b921b4d8d15e90a821e5f78cf9e2a2952c36f37efe692325e8be21ee693e WatchSource:0}: Error finding container 0791b921b4d8d15e90a821e5f78cf9e2a2952c36f37efe692325e8be21ee693e: Status 404 returned error can't find the container with id 0791b921b4d8d15e90a821e5f78cf9e2a2952c36f37efe692325e8be21ee693e Nov 03 16:20:44.288838 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:44.288802 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:44.288904 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:44.288871 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:20:45.168659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:45.168617 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" event={"ID":"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316","Type":"ContainerStarted","Data":"2ed31fe73909ff70aa1da897834cbd98e587d8682788dd240107d7f1d9a1c3b0"} Nov 03 16:20:45.169137 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:45.168674 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" event={"ID":"d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316","Type":"ContainerStarted","Data":"0791b921b4d8d15e90a821e5f78cf9e2a2952c36f37efe692325e8be21ee693e"} Nov 03 16:20:45.169137 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:45.168832 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:45.170491 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:45.170466 4947 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zp4kt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:20:45.170631 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:45.170518 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" podUID="d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:20:45.233067 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:45.233012 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" podStartSLOduration=2.23299613 podStartE2EDuration="2.23299613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:20:45.232935115 +0000 UTC m=+388.821935084" watchObservedRunningTime="2026-11-03 16:20:45.23299613 +0000 UTC m=+388.821996096" Nov 03 16:20:46.178215 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.178179 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zp4kt" Nov 03 16:20:46.278661 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.278630 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd"] Nov 03 16:20:46.278866 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.278846 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" podUID="938147b7-b53f-439b-a261-7d0d81dcf7bb" containerName="discovery" containerID="cri-o://152b82e2ab98c4bbc3963c895aeb892ffb57d05d9e8019b5d383358e925fd866" gracePeriod=30 Nov 03 16:20:46.526860 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.526832 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:20:46.592442 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.592409 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-cacerts\" (UniqueName: \"kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-cacerts\") pod \"938147b7-b53f-439b-a261-7d0d81dcf7bb\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " Nov 03 16:20:46.592442 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.592446 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938147b7-b53f-439b-a261-7d0d81dcf7bb-kube-api-access-j7xmm\" (UniqueName: \"kubernetes.io/projected/938147b7-b53f-439b-a261-7d0d81dcf7bb-kube-api-access-j7xmm\") pod \"938147b7-b53f-439b-a261-7d0d81dcf7bb\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " Nov 03 16:20:46.592715 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.592496 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/938147b7-b53f-439b-a261-7d0d81dcf7bb-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/938147b7-b53f-439b-a261-7d0d81dcf7bb-local-certs\") pod \"938147b7-b53f-439b-a261-7d0d81dcf7bb\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " Nov 03 16:20:46.592715 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.592533 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-csr-dns-cert\") pod \"938147b7-b53f-439b-a261-7d0d81dcf7bb\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " Nov 03 16:20:46.592715 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.592566 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-csr-ca-configmap\") pod \"938147b7-b53f-439b-a261-7d0d81dcf7bb\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " Nov 03 16:20:46.592715 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.592611 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-token\" (UniqueName: \"kubernetes.io/projected/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-token\") pod \"938147b7-b53f-439b-a261-7d0d81dcf7bb\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " Nov 03 16:20:46.592715 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.592668 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-kubeconfig\") pod \"938147b7-b53f-439b-a261-7d0d81dcf7bb\" (UID: \"938147b7-b53f-439b-a261-7d0d81dcf7bb\") " Nov 03 16:20:46.593812 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.593773 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-csr-ca-configmap" pod "938147b7-b53f-439b-a261-7d0d81dcf7bb" (UID: "938147b7-b53f-439b-a261-7d0d81dcf7bb"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:20:46.595612 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.595558 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-cacerts" pod "938147b7-b53f-439b-a261-7d0d81dcf7bb" (UID: "938147b7-b53f-439b-a261-7d0d81dcf7bb"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:20:46.596027 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.595980 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-csr-dns-cert" pod "938147b7-b53f-439b-a261-7d0d81dcf7bb" (UID: "938147b7-b53f-439b-a261-7d0d81dcf7bb"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:20:46.596272 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.596237 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-kubeconfig" pod "938147b7-b53f-439b-a261-7d0d81dcf7bb" (UID: "938147b7-b53f-439b-a261-7d0d81dcf7bb"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:20:46.596272 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.596243 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938147b7-b53f-439b-a261-7d0d81dcf7bb-kube-api-access-j7xmm" pod "938147b7-b53f-439b-a261-7d0d81dcf7bb" (UID: "938147b7-b53f-439b-a261-7d0d81dcf7bb"). InnerVolumeSpecName "kube-api-access-j7xmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:20:46.596272 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.596248 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/938147b7-b53f-439b-a261-7d0d81dcf7bb-local-certs" pod "938147b7-b53f-439b-a261-7d0d81dcf7bb" (UID: "938147b7-b53f-439b-a261-7d0d81dcf7bb"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:20:46.596583 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.596553 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-token" pod "938147b7-b53f-439b-a261-7d0d81dcf7bb" (UID: "938147b7-b53f-439b-a261-7d0d81dcf7bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:20:46.693900 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.693865 4947 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-cacerts\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:20:46.693900 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.693895 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7xmm\" (UniqueName: \"kubernetes.io/projected/938147b7-b53f-439b-a261-7d0d81dcf7bb-kube-api-access-j7xmm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:20:46.693900 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.693907 4947 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/938147b7-b53f-439b-a261-7d0d81dcf7bb-local-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:20:46.694145 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.693916 4947 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-csr-dns-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:20:46.694145 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.693924 4947 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-csr-ca-configmap\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:20:46.694145 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.693933 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:20:46.694145 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:46.693941 4947 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/938147b7-b53f-439b-a261-7d0d81dcf7bb-istio-kubeconfig\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:20:47.182254 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:47.182217 4947 generic.go:356] "Generic (PLEG): container finished" podID="938147b7-b53f-439b-a261-7d0d81dcf7bb" containerID="152b82e2ab98c4bbc3963c895aeb892ffb57d05d9e8019b5d383358e925fd866" exitCode=0 Nov 03 16:20:47.182796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:47.182270 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" Nov 03 16:20:47.182796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:47.182309 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" event={"ID":"938147b7-b53f-439b-a261-7d0d81dcf7bb","Type":"ContainerDied","Data":"152b82e2ab98c4bbc3963c895aeb892ffb57d05d9e8019b5d383358e925fd866"} Nov 03 16:20:47.182796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:47.182378 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd" event={"ID":"938147b7-b53f-439b-a261-7d0d81dcf7bb","Type":"ContainerDied","Data":"9085202ae8e522924560eec2d49d21a13c426906462d35017c6dab17427ec309"} Nov 03 16:20:47.182796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:47.182409 4947 scope.go:122] "RemoveContainer" containerID="152b82e2ab98c4bbc3963c895aeb892ffb57d05d9e8019b5d383358e925fd866" Nov 03 16:20:47.191431 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:47.191414 4947 scope.go:122] "RemoveContainer" containerID="152b82e2ab98c4bbc3963c895aeb892ffb57d05d9e8019b5d383358e925fd866" Nov 03 16:20:47.191763 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:20:47.191743 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"152b82e2ab98c4bbc3963c895aeb892ffb57d05d9e8019b5d383358e925fd866\": container with ID starting with 152b82e2ab98c4bbc3963c895aeb892ffb57d05d9e8019b5d383358e925fd866 not found: ID does not exist" containerID="152b82e2ab98c4bbc3963c895aeb892ffb57d05d9e8019b5d383358e925fd866" Nov 03 16:20:47.191840 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:47.191772 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152b82e2ab98c4bbc3963c895aeb892ffb57d05d9e8019b5d383358e925fd866"} err="failed to get container status \"152b82e2ab98c4bbc3963c895aeb892ffb57d05d9e8019b5d383358e925fd866\": rpc error: code = NotFound desc = could not find container \"152b82e2ab98c4bbc3963c895aeb892ffb57d05d9e8019b5d383358e925fd866\": container with ID starting with 152b82e2ab98c4bbc3963c895aeb892ffb57d05d9e8019b5d383358e925fd866 not found: ID does not exist" Nov 03 16:20:47.237647 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:47.237616 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd"] Nov 03 16:20:47.245130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:47.245105 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xclsd"] Nov 03 16:20:48.563260 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:20:48.563228 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="938147b7-b53f-439b-a261-7d0d81dcf7bb" path="/var/lib/kubelet/pods/938147b7-b53f-439b-a261-7d0d81dcf7bb/volumes" Nov 03 16:21:00.129298 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:21:00.129221 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:21:30.130034 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:21:30.129993 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:22:00.130742 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:22:00.130676 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:22:30.129760 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:22:30.129653 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:23:00.128557 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:23:00.128526 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:23:30.129149 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:23:30.129109 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:24:00.129366 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:24:00.129286 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:24:16.605829 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:24:16.605797 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:24:16.606633 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:24:16.606615 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:24:30.128488 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:24:30.128454 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:25:00.129466 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:25:00.129423 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:25:30.129192 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:25:30.129100 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:26:00.129411 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:26:00.129370 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:26:30.128485 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:26:30.128439 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:27:00.129645 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:27:00.129562 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:27:30.128769 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:27:30.128733 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:28:00.128640 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:28:00.128597 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:28:30.128117 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:28:30.128023 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:29:00.130104 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:29:00.130003 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:29:16.632012 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:29:16.631980 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:29:16.633472 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:29:16.633452 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:29:30.129293 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:29:30.129254 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:30:00.132440 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:30:00.132348 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:30:21.322363 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.322327 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-njkdc"] Nov 03 16:30:21.322788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.322662 4947 cpu_manager.go:414] "containerMap: removing container" podUID="938147b7-b53f-439b-a261-7d0d81dcf7bb" containerName="discovery" Nov 03 16:30:21.322788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.322673 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938147b7-b53f-439b-a261-7d0d81dcf7bb" containerName="discovery" Nov 03 16:30:21.322788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.322755 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="938147b7-b53f-439b-a261-7d0d81dcf7bb" containerName="discovery" Nov 03 16:30:21.325665 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.325649 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-njkdc" Nov 03 16:30:21.341861 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.341840 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 16:30:21.344460 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.344439 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 16:30:21.344569 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.344553 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 16:30:21.355506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.355476 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pj2d7\"" Nov 03 16:30:21.359163 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.359136 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-njkdc"] Nov 03 16:30:21.408798 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.408758 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9ec3c967-ab28-414a-a7fc-7e92b2861f25-config-file\") pod \"limitador-limitador-74dfcfb894-njkdc\" (UID: \"9ec3c967-ab28-414a-a7fc-7e92b2861f25\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-njkdc" Nov 03 16:30:21.408798 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.408790 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4x4m\" (UniqueName: \"kubernetes.io/projected/9ec3c967-ab28-414a-a7fc-7e92b2861f25-kube-api-access-r4x4m\") pod \"limitador-limitador-74dfcfb894-njkdc\" (UID: \"9ec3c967-ab28-414a-a7fc-7e92b2861f25\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-njkdc" Nov 03 16:30:21.464062 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.464027 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-njkdc"] Nov 03 16:30:21.510188 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.510146 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9ec3c967-ab28-414a-a7fc-7e92b2861f25-config-file\") pod \"limitador-limitador-74dfcfb894-njkdc\" (UID: \"9ec3c967-ab28-414a-a7fc-7e92b2861f25\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-njkdc" Nov 03 16:30:21.510188 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.510193 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4x4m\" (UniqueName: \"kubernetes.io/projected/9ec3c967-ab28-414a-a7fc-7e92b2861f25-kube-api-access-r4x4m\") pod \"limitador-limitador-74dfcfb894-njkdc\" (UID: \"9ec3c967-ab28-414a-a7fc-7e92b2861f25\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-njkdc" Nov 03 16:30:21.510798 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.510778 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9ec3c967-ab28-414a-a7fc-7e92b2861f25-config-file\") pod \"limitador-limitador-74dfcfb894-njkdc\" (UID: \"9ec3c967-ab28-414a-a7fc-7e92b2861f25\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-njkdc" Nov 03 16:30:21.535090 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.535058 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4x4m\" (UniqueName: \"kubernetes.io/projected/9ec3c967-ab28-414a-a7fc-7e92b2861f25-kube-api-access-r4x4m\") pod \"limitador-limitador-74dfcfb894-njkdc\" (UID: \"9ec3c967-ab28-414a-a7fc-7e92b2861f25\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-njkdc" Nov 03 16:30:21.638433 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.638344 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-njkdc" Nov 03 16:30:21.942955 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.942927 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-njkdc"] Nov 03 16:30:21.944745 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:30:21.944718 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ec3c967_ab28_414a_a7fc_7e92b2861f25.slice/crio-5d543a78eb6ff8544593f4d5555ec51b462eb23dd8de393e90b8910b4fe2c654 WatchSource:0}: Error finding container 5d543a78eb6ff8544593f4d5555ec51b462eb23dd8de393e90b8910b4fe2c654: Status 404 returned error can't find the container with id 5d543a78eb6ff8544593f4d5555ec51b462eb23dd8de393e90b8910b4fe2c654 Nov 03 16:30:21.946956 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:21.946936 4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:30:22.205263 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:22.205184 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:30:22.209687 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:22.209667 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-t6dn2" Nov 03 16:30:22.228889 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:22.228860 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:30:22.293618 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:22.293583 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:30:22.315858 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:22.315829 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m64pm\" (UniqueName: \"kubernetes.io/projected/879fa899-009b-44d6-adb2-ae2d9a82e427-kube-api-access-m64pm\") pod \"limitador-limitador-7c9b87d574-t6dn2\" (UID: \"879fa899-009b-44d6-adb2-ae2d9a82e427\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-t6dn2" Nov 03 16:30:22.315979 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:22.315875 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/879fa899-009b-44d6-adb2-ae2d9a82e427-config-file\") pod \"limitador-limitador-7c9b87d574-t6dn2\" (UID: \"879fa899-009b-44d6-adb2-ae2d9a82e427\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-t6dn2" Nov 03 16:30:22.416391 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:22.416352 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m64pm\" (UniqueName: \"kubernetes.io/projected/879fa899-009b-44d6-adb2-ae2d9a82e427-kube-api-access-m64pm\") pod \"limitador-limitador-7c9b87d574-t6dn2\" (UID: \"879fa899-009b-44d6-adb2-ae2d9a82e427\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-t6dn2" Nov 03 16:30:22.416832 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:22.416404 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/879fa899-009b-44d6-adb2-ae2d9a82e427-config-file\") pod \"limitador-limitador-7c9b87d574-t6dn2\" (UID: \"879fa899-009b-44d6-adb2-ae2d9a82e427\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-t6dn2" Nov 03 16:30:22.417053 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:22.417035 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/879fa899-009b-44d6-adb2-ae2d9a82e427-config-file\") pod \"limitador-limitador-7c9b87d574-t6dn2\" (UID: \"879fa899-009b-44d6-adb2-ae2d9a82e427\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-t6dn2" Nov 03 16:30:22.436006 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:22.435977 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-njkdc" event={"ID":"9ec3c967-ab28-414a-a7fc-7e92b2861f25","Type":"ContainerStarted","Data":"5d543a78eb6ff8544593f4d5555ec51b462eb23dd8de393e90b8910b4fe2c654"} Nov 03 16:30:22.441395 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:22.441367 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m64pm\" (UniqueName: \"kubernetes.io/projected/879fa899-009b-44d6-adb2-ae2d9a82e427-kube-api-access-m64pm\") pod \"limitador-limitador-7c9b87d574-t6dn2\" (UID: \"879fa899-009b-44d6-adb2-ae2d9a82e427\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-t6dn2" Nov 03 16:30:22.522801 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:22.522764 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-t6dn2" Nov 03 16:30:22.832529 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:22.832178 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:30:22.835314 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:30:22.835281 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879fa899_009b_44d6_adb2_ae2d9a82e427.slice/crio-5cfefa509c3571a593591863e1c4b0015723271af5b9bfd8d8070d4b01b7d71e WatchSource:0}: Error finding container 5cfefa509c3571a593591863e1c4b0015723271af5b9bfd8d8070d4b01b7d71e: Status 404 returned error can't find the container with id 5cfefa509c3571a593591863e1c4b0015723271af5b9bfd8d8070d4b01b7d71e Nov 03 16:30:23.445209 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:23.445130 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-t6dn2" event={"ID":"879fa899-009b-44d6-adb2-ae2d9a82e427","Type":"ContainerStarted","Data":"5cfefa509c3571a593591863e1c4b0015723271af5b9bfd8d8070d4b01b7d71e"} Nov 03 16:30:24.451578 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:24.451528 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-njkdc" event={"ID":"9ec3c967-ab28-414a-a7fc-7e92b2861f25","Type":"ContainerStarted","Data":"5ccc4400b9858855c2d7c83bf57957367a80cdba0226e09e0d53a2356def5b72"} Nov 03 16:30:24.452053 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:24.451625 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-njkdc" Nov 03 16:30:24.453181 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:24.453158 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-t6dn2" event={"ID":"879fa899-009b-44d6-adb2-ae2d9a82e427","Type":"ContainerStarted","Data":"55f94fcbadad7a78c9c94ee7da3d2dc09c963a08c6dd5476553694dc8fe37ad2"} Nov 03 16:30:24.453293 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:24.453281 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-t6dn2" Nov 03 16:30:24.496360 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:24.496311 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-njkdc" podStartSLOduration=1.152707175 podStartE2EDuration="3.496298845s" totalImagesPullingTime="2.34359167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:30:24.495398564 +0000 UTC m=+968.084398533" watchObservedRunningTime="2026-11-03 16:30:24.496298845 +0000 UTC m=+968.085298813" Nov 03 16:30:24.540288 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:24.540168 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-t6dn2" podStartSLOduration=1.049612203 podStartE2EDuration="2.540152096s" totalImagesPullingTime="1.490539893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:30:24.539887842 +0000 UTC m=+968.128887813" watchObservedRunningTime="2026-11-03 16:30:24.540152096 +0000 UTC m=+968.129152065" Nov 03 16:30:30.129643 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:30:30.129595 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:30:35.459262 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:35.459232 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-njkdc" Nov 03 16:30:35.459794 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:35.459306 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-t6dn2" Nov 03 16:30:35.578488 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:35.578453 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-njkdc"] Nov 03 16:30:35.578709 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:35.578633 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-njkdc" podUID="9ec3c967-ab28-414a-a7fc-7e92b2861f25" containerName="limitador" containerID="cri-o://5ccc4400b9858855c2d7c83bf57957367a80cdba0226e09e0d53a2356def5b72" gracePeriod=30 Nov 03 16:30:36.118338 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:36.118314 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-njkdc" Nov 03 16:30:36.137733 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:36.137643 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ec3c967-ab28-414a-a7fc-7e92b2861f25-config-file\" (UniqueName: \"kubernetes.io/configmap/9ec3c967-ab28-414a-a7fc-7e92b2861f25-config-file\") pod \"9ec3c967-ab28-414a-a7fc-7e92b2861f25\" (UID: \"9ec3c967-ab28-414a-a7fc-7e92b2861f25\") " Nov 03 16:30:36.138025 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:36.138001 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ec3c967-ab28-414a-a7fc-7e92b2861f25-config-file" pod "9ec3c967-ab28-414a-a7fc-7e92b2861f25" (UID: "9ec3c967-ab28-414a-a7fc-7e92b2861f25"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:30:36.238910 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:36.238877 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ec3c967-ab28-414a-a7fc-7e92b2861f25-kube-api-access-r4x4m\" (UniqueName: \"kubernetes.io/projected/9ec3c967-ab28-414a-a7fc-7e92b2861f25-kube-api-access-r4x4m\") pod \"9ec3c967-ab28-414a-a7fc-7e92b2861f25\" (UID: \"9ec3c967-ab28-414a-a7fc-7e92b2861f25\") " Nov 03 16:30:36.239082 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:36.239064 4947 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9ec3c967-ab28-414a-a7fc-7e92b2861f25-config-file\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:30:36.241143 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:36.241123 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ec3c967-ab28-414a-a7fc-7e92b2861f25-kube-api-access-r4x4m" pod "9ec3c967-ab28-414a-a7fc-7e92b2861f25" (UID: "9ec3c967-ab28-414a-a7fc-7e92b2861f25"). InnerVolumeSpecName "kube-api-access-r4x4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:30:36.340337 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:36.340305 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4x4m\" (UniqueName: \"kubernetes.io/projected/9ec3c967-ab28-414a-a7fc-7e92b2861f25-kube-api-access-r4x4m\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:30:36.505992 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:36.505956 4947 generic.go:356] "Generic (PLEG): container finished" podID="9ec3c967-ab28-414a-a7fc-7e92b2861f25" containerID="5ccc4400b9858855c2d7c83bf57957367a80cdba0226e09e0d53a2356def5b72" exitCode=0 Nov 03 16:30:36.506458 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:36.506024 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-njkdc" Nov 03 16:30:36.506458 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:36.506030 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-njkdc" event={"ID":"9ec3c967-ab28-414a-a7fc-7e92b2861f25","Type":"ContainerDied","Data":"5ccc4400b9858855c2d7c83bf57957367a80cdba0226e09e0d53a2356def5b72"} Nov 03 16:30:36.506458 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:36.506123 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-njkdc" event={"ID":"9ec3c967-ab28-414a-a7fc-7e92b2861f25","Type":"ContainerDied","Data":"5d543a78eb6ff8544593f4d5555ec51b462eb23dd8de393e90b8910b4fe2c654"} Nov 03 16:30:36.506458 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:36.506151 4947 scope.go:122] "RemoveContainer" containerID="5ccc4400b9858855c2d7c83bf57957367a80cdba0226e09e0d53a2356def5b72" Nov 03 16:30:36.515312 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:36.515288 4947 scope.go:122] "RemoveContainer" containerID="5ccc4400b9858855c2d7c83bf57957367a80cdba0226e09e0d53a2356def5b72" Nov 03 16:30:36.515625 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:30:36.515599 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ccc4400b9858855c2d7c83bf57957367a80cdba0226e09e0d53a2356def5b72\": container with ID starting with 5ccc4400b9858855c2d7c83bf57957367a80cdba0226e09e0d53a2356def5b72 not found: ID does not exist" containerID="5ccc4400b9858855c2d7c83bf57957367a80cdba0226e09e0d53a2356def5b72" Nov 03 16:30:36.515681 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:36.515636 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ccc4400b9858855c2d7c83bf57957367a80cdba0226e09e0d53a2356def5b72"} err="failed to get container status \"5ccc4400b9858855c2d7c83bf57957367a80cdba0226e09e0d53a2356def5b72\": rpc error: code = NotFound desc = could not find container \"5ccc4400b9858855c2d7c83bf57957367a80cdba0226e09e0d53a2356def5b72\": container with ID starting with 5ccc4400b9858855c2d7c83bf57957367a80cdba0226e09e0d53a2356def5b72 not found: ID does not exist" Nov 03 16:30:36.565239 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:36.565209 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-njkdc"] Nov 03 16:30:36.573551 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:36.573526 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-njkdc"] Nov 03 16:30:38.562837 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:38.562806 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ec3c967-ab28-414a-a7fc-7e92b2861f25" path="/var/lib/kubelet/pods/9ec3c967-ab28-414a-a7fc-7e92b2861f25/volumes" Nov 03 16:30:56.766096 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.766060 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58f54db7dc-wrcgq"] Nov 03 16:30:56.766876 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.766852 4947 cpu_manager.go:414] "containerMap: removing container" podUID="9ec3c967-ab28-414a-a7fc-7e92b2861f25" containerName="limitador" Nov 03 16:30:56.766876 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.766872 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ec3c967-ab28-414a-a7fc-7e92b2861f25" containerName="limitador" Nov 03 16:30:56.767025 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.766936 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ec3c967-ab28-414a-a7fc-7e92b2861f25" containerName="limitador" Nov 03 16:30:56.769378 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.769355 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f54db7dc-wrcgq" Nov 03 16:30:56.786643 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.786620 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 16:30:56.799983 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.799955 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sgj9n\"" Nov 03 16:30:56.800182 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.800016 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 16:30:56.800182 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.800025 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 16:30:56.800278 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.800035 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 16:30:56.800278 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.800070 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 16:30:56.801653 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.801632 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58f54db7dc-wrcgq"] Nov 03 16:30:56.809478 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.809454 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-metrics-certs\") pod \"maas-controller-58f54db7dc-wrcgq\" (UID: \"324bbb89-74f5-4d9d-ab15-c3c801e4cb18\") " pod="opendatahub/maas-controller-58f54db7dc-wrcgq" Nov 03 16:30:56.809616 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.809596 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-webhook-cert\") pod \"maas-controller-58f54db7dc-wrcgq\" (UID: \"324bbb89-74f5-4d9d-ab15-c3c801e4cb18\") " pod="opendatahub/maas-controller-58f54db7dc-wrcgq" Nov 03 16:30:56.809657 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.809624 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x979t\" (UniqueName: \"kubernetes.io/projected/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-kube-api-access-x979t\") pod \"maas-controller-58f54db7dc-wrcgq\" (UID: \"324bbb89-74f5-4d9d-ab15-c3c801e4cb18\") " pod="opendatahub/maas-controller-58f54db7dc-wrcgq" Nov 03 16:30:56.911065 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.911016 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-webhook-cert\") pod \"maas-controller-58f54db7dc-wrcgq\" (UID: \"324bbb89-74f5-4d9d-ab15-c3c801e4cb18\") " pod="opendatahub/maas-controller-58f54db7dc-wrcgq" Nov 03 16:30:56.911065 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.911063 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x979t\" (UniqueName: \"kubernetes.io/projected/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-kube-api-access-x979t\") pod \"maas-controller-58f54db7dc-wrcgq\" (UID: \"324bbb89-74f5-4d9d-ab15-c3c801e4cb18\") " pod="opendatahub/maas-controller-58f54db7dc-wrcgq" Nov 03 16:30:56.911338 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.911099 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-metrics-certs\") pod \"maas-controller-58f54db7dc-wrcgq\" (UID: \"324bbb89-74f5-4d9d-ab15-c3c801e4cb18\") " pod="opendatahub/maas-controller-58f54db7dc-wrcgq" Nov 03 16:30:56.913794 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.913770 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-metrics-certs\") pod \"maas-controller-58f54db7dc-wrcgq\" (UID: \"324bbb89-74f5-4d9d-ab15-c3c801e4cb18\") " pod="opendatahub/maas-controller-58f54db7dc-wrcgq" Nov 03 16:30:56.913911 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.913875 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-webhook-cert\") pod \"maas-controller-58f54db7dc-wrcgq\" (UID: \"324bbb89-74f5-4d9d-ab15-c3c801e4cb18\") " pod="opendatahub/maas-controller-58f54db7dc-wrcgq" Nov 03 16:30:56.936802 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:56.936772 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x979t\" (UniqueName: \"kubernetes.io/projected/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-kube-api-access-x979t\") pod \"maas-controller-58f54db7dc-wrcgq\" (UID: \"324bbb89-74f5-4d9d-ab15-c3c801e4cb18\") " pod="opendatahub/maas-controller-58f54db7dc-wrcgq" Nov 03 16:30:57.081179 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:57.081097 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f54db7dc-wrcgq" Nov 03 16:30:57.280252 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:57.280222 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-54fd7b574c-bvhhh"] Nov 03 16:30:57.282538 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:57.282522 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd7b574c-bvhhh" Nov 03 16:30:57.306653 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:57.306625 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54fd7b574c-bvhhh"] Nov 03 16:30:57.383487 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:57.383466 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58f54db7dc-wrcgq"] Nov 03 16:30:57.385288 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:30:57.385264 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324bbb89_74f5_4d9d_ab15_c3c801e4cb18.slice/crio-5d68a7b988e9d3eda1cf1e177a160e1ced8908654548adc3071ec0cca3ae716f WatchSource:0}: Error finding container 5d68a7b988e9d3eda1cf1e177a160e1ced8908654548adc3071ec0cca3ae716f: Status 404 returned error can't find the container with id 5d68a7b988e9d3eda1cf1e177a160e1ced8908654548adc3071ec0cca3ae716f Nov 03 16:30:57.415518 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:57.415477 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a25c94-f899-43b0-89b4-d37d1fd5a364-metrics-certs\") pod \"maas-controller-54fd7b574c-bvhhh\" (UID: \"07a25c94-f899-43b0-89b4-d37d1fd5a364\") " pod="opendatahub/maas-controller-54fd7b574c-bvhhh" Nov 03 16:30:57.415688 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:57.415527 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07a25c94-f899-43b0-89b4-d37d1fd5a364-webhook-cert\") pod \"maas-controller-54fd7b574c-bvhhh\" (UID: \"07a25c94-f899-43b0-89b4-d37d1fd5a364\") " pod="opendatahub/maas-controller-54fd7b574c-bvhhh" Nov 03 16:30:57.415688 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:57.415657 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2l6w\" (UniqueName: \"kubernetes.io/projected/07a25c94-f899-43b0-89b4-d37d1fd5a364-kube-api-access-v2l6w\") pod \"maas-controller-54fd7b574c-bvhhh\" (UID: \"07a25c94-f899-43b0-89b4-d37d1fd5a364\") " pod="opendatahub/maas-controller-54fd7b574c-bvhhh" Nov 03 16:30:57.516821 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:57.516778 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2l6w\" (UniqueName: \"kubernetes.io/projected/07a25c94-f899-43b0-89b4-d37d1fd5a364-kube-api-access-v2l6w\") pod \"maas-controller-54fd7b574c-bvhhh\" (UID: \"07a25c94-f899-43b0-89b4-d37d1fd5a364\") " pod="opendatahub/maas-controller-54fd7b574c-bvhhh" Nov 03 16:30:57.516993 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:57.516867 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a25c94-f899-43b0-89b4-d37d1fd5a364-metrics-certs\") pod \"maas-controller-54fd7b574c-bvhhh\" (UID: \"07a25c94-f899-43b0-89b4-d37d1fd5a364\") " pod="opendatahub/maas-controller-54fd7b574c-bvhhh" Nov 03 16:30:57.516993 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:57.516907 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07a25c94-f899-43b0-89b4-d37d1fd5a364-webhook-cert\") pod \"maas-controller-54fd7b574c-bvhhh\" (UID: \"07a25c94-f899-43b0-89b4-d37d1fd5a364\") " pod="opendatahub/maas-controller-54fd7b574c-bvhhh" Nov 03 16:30:57.519457 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:57.519433 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07a25c94-f899-43b0-89b4-d37d1fd5a364-webhook-cert\") pod \"maas-controller-54fd7b574c-bvhhh\" (UID: \"07a25c94-f899-43b0-89b4-d37d1fd5a364\") " pod="opendatahub/maas-controller-54fd7b574c-bvhhh" Nov 03 16:30:57.519457 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:57.519447 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a25c94-f899-43b0-89b4-d37d1fd5a364-metrics-certs\") pod \"maas-controller-54fd7b574c-bvhhh\" (UID: \"07a25c94-f899-43b0-89b4-d37d1fd5a364\") " pod="opendatahub/maas-controller-54fd7b574c-bvhhh" Nov 03 16:30:57.541552 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:57.541529 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2l6w\" (UniqueName: \"kubernetes.io/projected/07a25c94-f899-43b0-89b4-d37d1fd5a364-kube-api-access-v2l6w\") pod \"maas-controller-54fd7b574c-bvhhh\" (UID: \"07a25c94-f899-43b0-89b4-d37d1fd5a364\") " pod="opendatahub/maas-controller-54fd7b574c-bvhhh" Nov 03 16:30:57.590257 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:57.590225 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f54db7dc-wrcgq" event={"ID":"324bbb89-74f5-4d9d-ab15-c3c801e4cb18","Type":"ContainerStarted","Data":"5d68a7b988e9d3eda1cf1e177a160e1ced8908654548adc3071ec0cca3ae716f"} Nov 03 16:30:57.594408 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:57.594392 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd7b574c-bvhhh" Nov 03 16:30:57.896194 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:57.896171 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54fd7b574c-bvhhh"] Nov 03 16:30:57.897961 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:30:57.897926 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a25c94_f899_43b0_89b4_d37d1fd5a364.slice/crio-1d3abe78eb1198f042f9d6e3af820182118414ce89c04924f0999e621386c796 WatchSource:0}: Error finding container 1d3abe78eb1198f042f9d6e3af820182118414ce89c04924f0999e621386c796: Status 404 returned error can't find the container with id 1d3abe78eb1198f042f9d6e3af820182118414ce89c04924f0999e621386c796 Nov 03 16:30:58.603396 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:30:58.603308 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd7b574c-bvhhh" event={"ID":"07a25c94-f899-43b0-89b4-d37d1fd5a364","Type":"ContainerStarted","Data":"1d3abe78eb1198f042f9d6e3af820182118414ce89c04924f0999e621386c796"} Nov 03 16:31:00.138915 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:31:00.138879 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:31:00.614725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:00.614636 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd7b574c-bvhhh" event={"ID":"07a25c94-f899-43b0-89b4-d37d1fd5a364","Type":"ContainerStarted","Data":"9c6fca25f979a0e22f4484f55494ee95049938af84a7b3807e231e0362a0cbc1"} Nov 03 16:31:00.614928 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:00.614764 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-54fd7b574c-bvhhh" Nov 03 16:31:00.616546 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:00.616515 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f54db7dc-wrcgq" event={"ID":"324bbb89-74f5-4d9d-ab15-c3c801e4cb18","Type":"ContainerStarted","Data":"29f8780e10d677030da53126c92614c3191cbdfb71f13db099ea6c0c0c21c8c4"} Nov 03 16:31:00.616742 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:00.616571 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58f54db7dc-wrcgq" Nov 03 16:31:00.672097 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:00.672044 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-54fd7b574c-bvhhh" podStartSLOduration=1.386148804 podStartE2EDuration="3.672029205s" totalImagesPullingTime="2.285880401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:31:00.660640948 +0000 UTC m=+1004.249640916" watchObservedRunningTime="2026-11-03 16:31:00.672029205 +0000 UTC m=+1004.261029248" Nov 03 16:31:00.709891 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:00.709825 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-58f54db7dc-wrcgq" podStartSLOduration=2.035806022 podStartE2EDuration="4.709804679s" totalImagesPullingTime="2.673998657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:31:00.707980474 +0000 UTC m=+1004.296980442" watchObservedRunningTime="2026-11-03 16:31:00.709804679 +0000 UTC m=+1004.298804649" Nov 03 16:31:02.160238 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.160204 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck"] Nov 03 16:31:02.163554 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.163537 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" Nov 03 16:31:02.185065 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.185036 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-5jn4t\"" Nov 03 16:31:02.187626 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.187603 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 16:31:02.198744 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.198713 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 16:31:02.198887 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.198760 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 16:31:02.200245 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.200214 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck"] Nov 03 16:31:02.259872 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.259838 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f54e03f-0ad5-4368-a9eb-78fad1b70156-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4grck\" (UID: \"9f54e03f-0ad5-4368-a9eb-78fad1b70156\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" Nov 03 16:31:02.260044 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.259880 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f54e03f-0ad5-4368-a9eb-78fad1b70156-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4grck\" (UID: \"9f54e03f-0ad5-4368-a9eb-78fad1b70156\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" Nov 03 16:31:02.260044 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.260009 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mst98\" (UniqueName: \"kubernetes.io/projected/9f54e03f-0ad5-4368-a9eb-78fad1b70156-kube-api-access-mst98\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4grck\" (UID: \"9f54e03f-0ad5-4368-a9eb-78fad1b70156\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" Nov 03 16:31:02.260115 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.260045 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9f54e03f-0ad5-4368-a9eb-78fad1b70156-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4grck\" (UID: \"9f54e03f-0ad5-4368-a9eb-78fad1b70156\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" Nov 03 16:31:02.360622 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.360583 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f54e03f-0ad5-4368-a9eb-78fad1b70156-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4grck\" (UID: \"9f54e03f-0ad5-4368-a9eb-78fad1b70156\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" Nov 03 16:31:02.360877 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.360633 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mst98\" (UniqueName: \"kubernetes.io/projected/9f54e03f-0ad5-4368-a9eb-78fad1b70156-kube-api-access-mst98\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4grck\" (UID: \"9f54e03f-0ad5-4368-a9eb-78fad1b70156\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" Nov 03 16:31:02.360877 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.360752 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9f54e03f-0ad5-4368-a9eb-78fad1b70156-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4grck\" (UID: \"9f54e03f-0ad5-4368-a9eb-78fad1b70156\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" Nov 03 16:31:02.360989 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.360879 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f54e03f-0ad5-4368-a9eb-78fad1b70156-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4grck\" (UID: \"9f54e03f-0ad5-4368-a9eb-78fad1b70156\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" Nov 03 16:31:02.361551 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.361473 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9f54e03f-0ad5-4368-a9eb-78fad1b70156-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4grck\" (UID: \"9f54e03f-0ad5-4368-a9eb-78fad1b70156\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" Nov 03 16:31:02.361739 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.361662 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/9f54e03f-0ad5-4368-a9eb-78fad1b70156-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4grck\" (UID: \"9f54e03f-0ad5-4368-a9eb-78fad1b70156\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" Nov 03 16:31:02.363289 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.363268 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9f54e03f-0ad5-4368-a9eb-78fad1b70156-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4grck\" (UID: \"9f54e03f-0ad5-4368-a9eb-78fad1b70156\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" Nov 03 16:31:02.385475 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.385447 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mst98\" (UniqueName: \"kubernetes.io/projected/9f54e03f-0ad5-4368-a9eb-78fad1b70156-kube-api-access-mst98\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4grck\" (UID: \"9f54e03f-0ad5-4368-a9eb-78fad1b70156\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" Nov 03 16:31:02.476823 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.476693 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" Nov 03 16:31:02.778079 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:02.778056 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck"] Nov 03 16:31:02.780063 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:31:02.780033 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f54e03f_0ad5_4368_a9eb_78fad1b70156.slice/crio-f9a053b017df9b92349fa64123f5e885e04cee2735068d51286c32f2148d203e WatchSource:0}: Error finding container f9a053b017df9b92349fa64123f5e885e04cee2735068d51286c32f2148d203e: Status 404 returned error can't find the container with id f9a053b017df9b92349fa64123f5e885e04cee2735068d51286c32f2148d203e Nov 03 16:31:03.632076 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:03.632043 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" event={"ID":"9f54e03f-0ad5-4368-a9eb-78fad1b70156","Type":"ContainerStarted","Data":"f9a053b017df9b92349fa64123f5e885e04cee2735068d51286c32f2148d203e"} Nov 03 16:31:11.624619 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:11.624586 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-54fd7b574c-bvhhh" Nov 03 16:31:11.625237 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:11.624687 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58f54db7dc-wrcgq" Nov 03 16:31:11.750273 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:11.750238 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58f54db7dc-wrcgq"] Nov 03 16:31:11.750564 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:11.750504 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-58f54db7dc-wrcgq" podUID="324bbb89-74f5-4d9d-ab15-c3c801e4cb18" containerName="manager" containerID="cri-o://29f8780e10d677030da53126c92614c3191cbdfb71f13db099ea6c0c0c21c8c4" gracePeriod=30 Nov 03 16:31:12.017119 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.017092 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f54db7dc-wrcgq" Nov 03 16:31:12.151585 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.151550 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-metrics-certs\" (UniqueName: \"kubernetes.io/secret/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-metrics-certs\") pod \"324bbb89-74f5-4d9d-ab15-c3c801e4cb18\" (UID: \"324bbb89-74f5-4d9d-ab15-c3c801e4cb18\") " Nov 03 16:31:12.151804 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.151635 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-kube-api-access-x979t\" (UniqueName: \"kubernetes.io/projected/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-kube-api-access-x979t\") pod \"324bbb89-74f5-4d9d-ab15-c3c801e4cb18\" (UID: \"324bbb89-74f5-4d9d-ab15-c3c801e4cb18\") " Nov 03 16:31:12.151804 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.151725 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-webhook-cert\" (UniqueName: \"kubernetes.io/secret/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-webhook-cert\") pod \"324bbb89-74f5-4d9d-ab15-c3c801e4cb18\" (UID: \"324bbb89-74f5-4d9d-ab15-c3c801e4cb18\") " Nov 03 16:31:12.154305 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.154270 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-webhook-cert" pod "324bbb89-74f5-4d9d-ab15-c3c801e4cb18" (UID: "324bbb89-74f5-4d9d-ab15-c3c801e4cb18"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:31:12.154444 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.154407 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-kube-api-access-x979t" pod "324bbb89-74f5-4d9d-ab15-c3c801e4cb18" (UID: "324bbb89-74f5-4d9d-ab15-c3c801e4cb18"). InnerVolumeSpecName "kube-api-access-x979t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:31:12.154483 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.154410 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-metrics-certs" pod "324bbb89-74f5-4d9d-ab15-c3c801e4cb18" (UID: "324bbb89-74f5-4d9d-ab15-c3c801e4cb18"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:31:12.252774 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.252741 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x979t\" (UniqueName: \"kubernetes.io/projected/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-kube-api-access-x979t\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:31:12.252774 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.252770 4947 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-webhook-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:31:12.252774 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.252781 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/324bbb89-74f5-4d9d-ab15-c3c801e4cb18-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:31:12.699881 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.699835 4947 generic.go:356] "Generic (PLEG): container finished" podID="324bbb89-74f5-4d9d-ab15-c3c801e4cb18" containerID="29f8780e10d677030da53126c92614c3191cbdfb71f13db099ea6c0c0c21c8c4" exitCode=0 Nov 03 16:31:12.700301 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.699907 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58f54db7dc-wrcgq" Nov 03 16:31:12.700301 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.699918 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f54db7dc-wrcgq" event={"ID":"324bbb89-74f5-4d9d-ab15-c3c801e4cb18","Type":"ContainerDied","Data":"29f8780e10d677030da53126c92614c3191cbdfb71f13db099ea6c0c0c21c8c4"} Nov 03 16:31:12.700301 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.699975 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58f54db7dc-wrcgq" event={"ID":"324bbb89-74f5-4d9d-ab15-c3c801e4cb18","Type":"ContainerDied","Data":"5d68a7b988e9d3eda1cf1e177a160e1ced8908654548adc3071ec0cca3ae716f"} Nov 03 16:31:12.700301 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.699998 4947 scope.go:122] "RemoveContainer" containerID="29f8780e10d677030da53126c92614c3191cbdfb71f13db099ea6c0c0c21c8c4" Nov 03 16:31:12.709984 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.709957 4947 scope.go:122] "RemoveContainer" containerID="29f8780e10d677030da53126c92614c3191cbdfb71f13db099ea6c0c0c21c8c4" Nov 03 16:31:12.710365 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:31:12.710340 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f8780e10d677030da53126c92614c3191cbdfb71f13db099ea6c0c0c21c8c4\": container with ID starting with 29f8780e10d677030da53126c92614c3191cbdfb71f13db099ea6c0c0c21c8c4 not found: ID does not exist" containerID="29f8780e10d677030da53126c92614c3191cbdfb71f13db099ea6c0c0c21c8c4" Nov 03 16:31:12.710473 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.710375 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f8780e10d677030da53126c92614c3191cbdfb71f13db099ea6c0c0c21c8c4"} err="failed to get container status \"29f8780e10d677030da53126c92614c3191cbdfb71f13db099ea6c0c0c21c8c4\": rpc error: code = NotFound desc = could not find container \"29f8780e10d677030da53126c92614c3191cbdfb71f13db099ea6c0c0c21c8c4\": container with ID starting with 29f8780e10d677030da53126c92614c3191cbdfb71f13db099ea6c0c0c21c8c4 not found: ID does not exist" Nov 03 16:31:12.755546 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.755511 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58f54db7dc-wrcgq"] Nov 03 16:31:12.769098 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:12.769050 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58f54db7dc-wrcgq"] Nov 03 16:31:14.562692 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:14.562660 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="324bbb89-74f5-4d9d-ab15-c3c801e4cb18" path="/var/lib/kubelet/pods/324bbb89-74f5-4d9d-ab15-c3c801e4cb18/volumes" Nov 03 16:31:14.711582 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:14.711500 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" event={"ID":"9f54e03f-0ad5-4368-a9eb-78fad1b70156","Type":"ContainerStarted","Data":"fd1463fcfac513e9acb896e81322f5043d82355808e5d547560c464b067d3a07"} Nov 03 16:31:14.711762 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:14.711680 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" Nov 03 16:31:14.756536 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:14.756488 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" podStartSLOduration=1.2132441840000001 podStartE2EDuration="12.756475231s" totalImagesPullingTime="11.543231047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:31:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:31:14.756320396 +0000 UTC m=+1018.345320366" watchObservedRunningTime="2026-11-03 16:31:14.756475231 +0000 UTC m=+1018.345475199" Nov 03 16:31:25.726178 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:25.726147 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4grck" Nov 03 16:31:30.130008 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:31:30.129924 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:31:31.346911 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.346877 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5"] Nov 03 16:31:31.347306 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.347212 4947 cpu_manager.go:414] "containerMap: removing container" podUID="324bbb89-74f5-4d9d-ab15-c3c801e4cb18" containerName="manager" Nov 03 16:31:31.347306 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.347222 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="324bbb89-74f5-4d9d-ab15-c3c801e4cb18" containerName="manager" Nov 03 16:31:31.347306 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.347273 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="324bbb89-74f5-4d9d-ab15-c3c801e4cb18" containerName="manager" Nov 03 16:31:31.354840 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.354823 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" Nov 03 16:31:31.386776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.386744 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 16:31:31.386962 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.386801 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 16:31:31.387307 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.387287 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 16:31:31.387423 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.387313 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 16:31:31.387423 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.387316 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-r7vlf\"" Nov 03 16:31:31.388025 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.387998 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5"] Nov 03 16:31:31.513838 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.513801 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-metrics-certs\") pod \"maas-api-6dfd88bddd-qkfs5\" (UID: \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" Nov 03 16:31:31.513838 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.513839 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2c98\" (UniqueName: \"kubernetes.io/projected/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-kube-api-access-c2c98\") pod \"maas-api-6dfd88bddd-qkfs5\" (UID: \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" Nov 03 16:31:31.514057 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.513858 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-openshift-service-ca\") pod \"maas-api-6dfd88bddd-qkfs5\" (UID: \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" Nov 03 16:31:31.514057 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.513978 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-maas-api-tls\") pod \"maas-api-6dfd88bddd-qkfs5\" (UID: \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" Nov 03 16:31:31.615054 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.614962 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-metrics-certs\") pod \"maas-api-6dfd88bddd-qkfs5\" (UID: \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" Nov 03 16:31:31.615054 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.615000 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2c98\" (UniqueName: \"kubernetes.io/projected/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-kube-api-access-c2c98\") pod \"maas-api-6dfd88bddd-qkfs5\" (UID: \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" Nov 03 16:31:31.615054 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.615021 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-openshift-service-ca\") pod \"maas-api-6dfd88bddd-qkfs5\" (UID: \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" Nov 03 16:31:31.615054 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.615048 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-maas-api-tls\") pod \"maas-api-6dfd88bddd-qkfs5\" (UID: \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" Nov 03 16:31:31.615801 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.615775 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-openshift-service-ca\") pod \"maas-api-6dfd88bddd-qkfs5\" (UID: \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" Nov 03 16:31:31.617793 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.617765 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-maas-api-tls\") pod \"maas-api-6dfd88bddd-qkfs5\" (UID: \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" Nov 03 16:31:31.617891 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.617834 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-metrics-certs\") pod \"maas-api-6dfd88bddd-qkfs5\" (UID: \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" Nov 03 16:31:31.639597 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.639569 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2c98\" (UniqueName: \"kubernetes.io/projected/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-kube-api-access-c2c98\") pod \"maas-api-6dfd88bddd-qkfs5\" (UID: \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\") " pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" Nov 03 16:31:31.668526 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.668476 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" Nov 03 16:31:31.971542 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:31.971513 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5"] Nov 03 16:31:31.973317 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:31:31.973295 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f49fb2_f64a_4ed4_b4b9_8f0585ad8e8a.slice/crio-16ad020f78ac05f7be07246189090671634c832a2230bd808368507c7397d66c WatchSource:0}: Error finding container 16ad020f78ac05f7be07246189090671634c832a2230bd808368507c7397d66c: Status 404 returned error can't find the container with id 16ad020f78ac05f7be07246189090671634c832a2230bd808368507c7397d66c Nov 03 16:31:32.786692 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:32.786654 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" event={"ID":"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a","Type":"ContainerStarted","Data":"16ad020f78ac05f7be07246189090671634c832a2230bd808368507c7397d66c"} Nov 03 16:31:33.007572 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:33.007530 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:31:33.792812 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:33.792774 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" event={"ID":"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a","Type":"ContainerStarted","Data":"00137f05e63d68ef82de20ff7b463c8850ba9ec92f7b6929d0804423023784b4"} Nov 03 16:31:33.793223 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:33.792918 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" Nov 03 16:31:33.839285 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:33.839227 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" podStartSLOduration=1.176508396 podStartE2EDuration="2.839211158s" totalImagesPullingTime="1.662702762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:31:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:31:33.838361142 +0000 UTC m=+1037.427361111" watchObservedRunningTime="2026-11-03 16:31:33.839211158 +0000 UTC m=+1037.428211128" Nov 03 16:31:39.802860 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:39.802830 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" Nov 03 16:31:40.799585 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:40.799523 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk"] Nov 03 16:31:40.805621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:40.805596 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" Nov 03 16:31:40.838387 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:40.838346 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk"] Nov 03 16:31:40.894384 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:40.894348 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9733bd5c-945d-4308-92b5-a838e82acf50-metrics-certs\") pod \"maas-api-74d6c75f5f-hwzwk\" (UID: \"9733bd5c-945d-4308-92b5-a838e82acf50\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" Nov 03 16:31:40.894562 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:40.894409 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9733bd5c-945d-4308-92b5-a838e82acf50-maas-api-tls\") pod \"maas-api-74d6c75f5f-hwzwk\" (UID: \"9733bd5c-945d-4308-92b5-a838e82acf50\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" Nov 03 16:31:40.894562 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:40.894469 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zrqv\" (UniqueName: \"kubernetes.io/projected/9733bd5c-945d-4308-92b5-a838e82acf50-kube-api-access-2zrqv\") pod \"maas-api-74d6c75f5f-hwzwk\" (UID: \"9733bd5c-945d-4308-92b5-a838e82acf50\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" Nov 03 16:31:40.894562 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:40.894549 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9733bd5c-945d-4308-92b5-a838e82acf50-openshift-service-ca\") pod \"maas-api-74d6c75f5f-hwzwk\" (UID: \"9733bd5c-945d-4308-92b5-a838e82acf50\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" Nov 03 16:31:40.995632 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:40.995592 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9733bd5c-945d-4308-92b5-a838e82acf50-maas-api-tls\") pod \"maas-api-74d6c75f5f-hwzwk\" (UID: \"9733bd5c-945d-4308-92b5-a838e82acf50\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" Nov 03 16:31:40.995632 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:40.995635 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zrqv\" (UniqueName: \"kubernetes.io/projected/9733bd5c-945d-4308-92b5-a838e82acf50-kube-api-access-2zrqv\") pod \"maas-api-74d6c75f5f-hwzwk\" (UID: \"9733bd5c-945d-4308-92b5-a838e82acf50\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" Nov 03 16:31:40.995934 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:40.995670 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9733bd5c-945d-4308-92b5-a838e82acf50-openshift-service-ca\") pod \"maas-api-74d6c75f5f-hwzwk\" (UID: \"9733bd5c-945d-4308-92b5-a838e82acf50\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" Nov 03 16:31:40.995934 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:40.995765 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9733bd5c-945d-4308-92b5-a838e82acf50-metrics-certs\") pod \"maas-api-74d6c75f5f-hwzwk\" (UID: \"9733bd5c-945d-4308-92b5-a838e82acf50\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" Nov 03 16:31:40.996274 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:40.996248 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9733bd5c-945d-4308-92b5-a838e82acf50-openshift-service-ca\") pod \"maas-api-74d6c75f5f-hwzwk\" (UID: \"9733bd5c-945d-4308-92b5-a838e82acf50\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" Nov 03 16:31:40.998371 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:40.998347 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9733bd5c-945d-4308-92b5-a838e82acf50-maas-api-tls\") pod \"maas-api-74d6c75f5f-hwzwk\" (UID: \"9733bd5c-945d-4308-92b5-a838e82acf50\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" Nov 03 16:31:40.998568 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:40.998549 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9733bd5c-945d-4308-92b5-a838e82acf50-metrics-certs\") pod \"maas-api-74d6c75f5f-hwzwk\" (UID: \"9733bd5c-945d-4308-92b5-a838e82acf50\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" Nov 03 16:31:41.021521 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:41.021486 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zrqv\" (UniqueName: \"kubernetes.io/projected/9733bd5c-945d-4308-92b5-a838e82acf50-kube-api-access-2zrqv\") pod \"maas-api-74d6c75f5f-hwzwk\" (UID: \"9733bd5c-945d-4308-92b5-a838e82acf50\") " pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" Nov 03 16:31:41.119137 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:41.119025 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" Nov 03 16:31:41.432467 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:41.432440 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk"] Nov 03 16:31:41.434562 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:31:41.434537 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9733bd5c_945d_4308_92b5_a838e82acf50.slice/crio-7f739a25b60f8caebf993097fe6cafb1e1253ccb22b25cd67bcc2338e469ba47 WatchSource:0}: Error finding container 7f739a25b60f8caebf993097fe6cafb1e1253ccb22b25cd67bcc2338e469ba47: Status 404 returned error can't find the container with id 7f739a25b60f8caebf993097fe6cafb1e1253ccb22b25cd67bcc2338e469ba47 Nov 03 16:31:41.828894 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:41.828856 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" event={"ID":"9733bd5c-945d-4308-92b5-a838e82acf50","Type":"ContainerStarted","Data":"7f739a25b60f8caebf993097fe6cafb1e1253ccb22b25cd67bcc2338e469ba47"} Nov 03 16:31:42.841273 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:42.841230 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" event={"ID":"9733bd5c-945d-4308-92b5-a838e82acf50","Type":"ContainerStarted","Data":"4f9640d12ee0c8b6b77c2ce4c2b84fa87169cf2a093680330b8358e14a37a7da"} Nov 03 16:31:42.841785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:42.841340 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" Nov 03 16:31:42.889816 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:42.889764 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" podStartSLOduration=2.21080839 podStartE2EDuration="2.889749115s" totalImagesPullingTime="678.940725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:31:42.889286008 +0000 UTC m=+1046.478285977" watchObservedRunningTime="2026-11-03 16:31:42.889749115 +0000 UTC m=+1046.478749084" Nov 03 16:31:48.850764 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:48.850730 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" Nov 03 16:31:48.932498 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:48.932461 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5"] Nov 03 16:31:48.932732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:48.932683 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" podUID="b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a" containerName="maas-api" containerID="cri-o://00137f05e63d68ef82de20ff7b463c8850ba9ec92f7b6929d0804423023784b4" gracePeriod=30 Nov 03 16:31:49.181495 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.181467 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" Nov 03 16:31:49.263550 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.263511 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-kube-api-access-c2c98\" (UniqueName: \"kubernetes.io/projected/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-kube-api-access-c2c98\") pod \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\" (UID: \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\") " Nov 03 16:31:49.263754 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.263563 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-metrics-certs\") pod \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\" (UID: \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\") " Nov 03 16:31:49.263754 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.263591 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-openshift-service-ca\") pod \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\" (UID: \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\") " Nov 03 16:31:49.263857 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.263782 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-maas-api-tls\") pod \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\" (UID: \"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a\") " Nov 03 16:31:49.263965 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.263943 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-openshift-service-ca" pod "b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a" (UID: "b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:31:49.264113 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.264095 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:31:49.266026 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.265999 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-metrics-certs" pod "b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a" (UID: "b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:31:49.266026 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.266014 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-maas-api-tls" pod "b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a" (UID: "b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:31:49.266026 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.265995 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-kube-api-access-c2c98" pod "b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a" (UID: "b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a"). InnerVolumeSpecName "kube-api-access-c2c98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:31:49.364527 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.364439 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2c98\" (UniqueName: \"kubernetes.io/projected/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-kube-api-access-c2c98\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:31:49.364527 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.364471 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:31:49.364527 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.364481 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:31:49.870643 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.870606 4947 generic.go:356] "Generic (PLEG): container finished" podID="b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a" containerID="00137f05e63d68ef82de20ff7b463c8850ba9ec92f7b6929d0804423023784b4" exitCode=0 Nov 03 16:31:49.871168 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.870674 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" Nov 03 16:31:49.871168 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.870680 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" event={"ID":"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a","Type":"ContainerDied","Data":"00137f05e63d68ef82de20ff7b463c8850ba9ec92f7b6929d0804423023784b4"} Nov 03 16:31:49.871168 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.870826 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5" event={"ID":"b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a","Type":"ContainerDied","Data":"16ad020f78ac05f7be07246189090671634c832a2230bd808368507c7397d66c"} Nov 03 16:31:49.871168 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.870848 4947 scope.go:122] "RemoveContainer" containerID="00137f05e63d68ef82de20ff7b463c8850ba9ec92f7b6929d0804423023784b4" Nov 03 16:31:49.879521 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.879503 4947 scope.go:122] "RemoveContainer" containerID="00137f05e63d68ef82de20ff7b463c8850ba9ec92f7b6929d0804423023784b4" Nov 03 16:31:49.879815 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:31:49.879797 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00137f05e63d68ef82de20ff7b463c8850ba9ec92f7b6929d0804423023784b4\": container with ID starting with 00137f05e63d68ef82de20ff7b463c8850ba9ec92f7b6929d0804423023784b4 not found: ID does not exist" containerID="00137f05e63d68ef82de20ff7b463c8850ba9ec92f7b6929d0804423023784b4" Nov 03 16:31:49.879913 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.879823 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00137f05e63d68ef82de20ff7b463c8850ba9ec92f7b6929d0804423023784b4"} err="failed to get container status \"00137f05e63d68ef82de20ff7b463c8850ba9ec92f7b6929d0804423023784b4\": rpc error: code = NotFound desc = could not find container \"00137f05e63d68ef82de20ff7b463c8850ba9ec92f7b6929d0804423023784b4\": container with ID starting with 00137f05e63d68ef82de20ff7b463c8850ba9ec92f7b6929d0804423023784b4 not found: ID does not exist" Nov 03 16:31:49.927765 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.927730 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5"] Nov 03 16:31:49.939241 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:49.939212 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6dfd88bddd-qkfs5"] Nov 03 16:31:50.563377 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:31:50.563337 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a" path="/var/lib/kubelet/pods/b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a/volumes" Nov 03 16:32:00.128651 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:32:00.128614 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:32:08.528729 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.528656 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689"] Nov 03 16:32:08.529239 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.529190 4947 cpu_manager.go:414] "containerMap: removing container" podUID="b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a" containerName="maas-api" Nov 03 16:32:08.529239 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.529208 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a" containerName="maas-api" Nov 03 16:32:08.529356 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.529321 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2f49fb2-f64a-4ed4-b4b9-8f0585ad8e8a" containerName="maas-api" Nov 03 16:32:08.531319 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.531300 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.565506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.565476 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689"] Nov 03 16:32:08.635076 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.635044 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.635076 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.635077 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.635332 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.635096 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.635332 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.635115 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.635332 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.635230 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.635332 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.635271 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.635332 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.635298 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.635600 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.635402 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.635600 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.635442 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw686\" (UniqueName: \"kubernetes.io/projected/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-kube-api-access-rw686\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.637906 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.637888 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9tjmt\"" Nov 03 16:32:08.737041 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.737008 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.737209 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.737053 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.737209 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.737099 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.737209 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.737128 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw686\" (UniqueName: \"kubernetes.io/projected/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-kube-api-access-rw686\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.737372 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.737219 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.737372 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.737244 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.737372 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.737285 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.737372 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.737305 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.737372 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.737338 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.737634 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.737486 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.737753 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.737735 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.737900 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.737876 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.737993 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.737933 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.738384 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.738352 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.739818 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.739795 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.740075 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.740054 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.762215 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.762183 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.762399 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.762380 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw686\" (UniqueName: \"kubernetes.io/projected/43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb-kube-api-access-rw686\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-rp689\" (UID: \"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:08.842760 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:08.842667 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:09.157926 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:09.157888 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689"] Nov 03 16:32:09.957235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:09.957190 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" event={"ID":"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb","Type":"ContainerStarted","Data":"972c14f54c39c423c071452d99dfc37159c166e440a0ac18d60dfc94715cfb68"} Nov 03 16:32:11.803326 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:11.803297 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:32:11.803582 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:11.803363 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:32:11.803582 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:11.803414 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:32:11.969307 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:11.969265 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" event={"ID":"43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb","Type":"ContainerStarted","Data":"75ae8212f11b74482a072ff395db818084e728fe1a6fc1fdb5c90b78997b84e4"} Nov 03 16:32:12.032863 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:12.032799 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" podStartSLOduration=1.392243908 podStartE2EDuration="4.032783751s" totalImagesPullingTime="2.640539843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:12.03153126 +0000 UTC m=+1075.620531232" watchObservedRunningTime="2026-11-03 16:32:12.032783751 +0000 UTC m=+1075.621783720" Nov 03 16:32:12.843206 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:12.843168 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:12.844819 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:12.844793 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rp689 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 16:32:12.844957 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:12.844849 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" podUID="43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 16:32:13.843670 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:13.843634 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rp689 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 16:32:13.844117 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:13.843687 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" podUID="43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 16:32:14.843394 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:14.843360 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rp689 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 16:32:14.843597 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:14.843416 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" podUID="43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 16:32:15.843260 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:15.843218 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rp689 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 16:32:15.843676 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:15.843295 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" podUID="43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 16:32:16.843715 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:16.843650 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rp689 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 16:32:16.844116 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:16.843735 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" podUID="43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 16:32:17.860320 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:17.860289 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:17.860848 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:17.860710 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:17.861688 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:17.861669 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" Nov 03 16:32:18.510451 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:18.510417 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-2lqnz"] Nov 03 16:32:18.513675 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:18.513658 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-2lqnz" Nov 03 16:32:18.545297 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:18.545269 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 16:32:18.545297 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:18.545279 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 16:32:18.545540 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:18.545350 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-w5jsc\"" Nov 03 16:32:18.546667 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:18.546646 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-2lqnz"] Nov 03 16:32:18.643678 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:18.643645 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a305a949-c2d0-45db-ad70-7d19c6d6fbde-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-2lqnz\" (UID: \"a305a949-c2d0-45db-ad70-7d19c6d6fbde\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-2lqnz" Nov 03 16:32:18.643678 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:18.643683 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpht2\" (UniqueName: \"kubernetes.io/projected/a305a949-c2d0-45db-ad70-7d19c6d6fbde-kube-api-access-mpht2\") pod \"ai-gateway-controller-5d8dddb676-2lqnz\" (UID: \"a305a949-c2d0-45db-ad70-7d19c6d6fbde\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-2lqnz" Nov 03 16:32:18.744468 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:18.744429 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a305a949-c2d0-45db-ad70-7d19c6d6fbde-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-2lqnz\" (UID: \"a305a949-c2d0-45db-ad70-7d19c6d6fbde\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-2lqnz" Nov 03 16:32:18.744468 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:18.744468 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpht2\" (UniqueName: \"kubernetes.io/projected/a305a949-c2d0-45db-ad70-7d19c6d6fbde-kube-api-access-mpht2\") pod \"ai-gateway-controller-5d8dddb676-2lqnz\" (UID: \"a305a949-c2d0-45db-ad70-7d19c6d6fbde\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-2lqnz" Nov 03 16:32:18.747192 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:18.747165 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a305a949-c2d0-45db-ad70-7d19c6d6fbde-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-2lqnz\" (UID: \"a305a949-c2d0-45db-ad70-7d19c6d6fbde\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-2lqnz" Nov 03 16:32:18.770062 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:18.769980 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpht2\" (UniqueName: \"kubernetes.io/projected/a305a949-c2d0-45db-ad70-7d19c6d6fbde-kube-api-access-mpht2\") pod \"ai-gateway-controller-5d8dddb676-2lqnz\" (UID: \"a305a949-c2d0-45db-ad70-7d19c6d6fbde\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-2lqnz" Nov 03 16:32:18.825692 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:18.825652 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-2lqnz" Nov 03 16:32:19.125414 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:19.125388 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-2lqnz"] Nov 03 16:32:19.127053 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:32:19.127026 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda305a949_c2d0_45db_ad70_7d19c6d6fbde.slice/crio-f49958cd11ab8df97b263fb6e0f0b7e2394fe56a198e3e68be146ccb53e84cd3 WatchSource:0}: Error finding container f49958cd11ab8df97b263fb6e0f0b7e2394fe56a198e3e68be146ccb53e84cd3: Status 404 returned error can't find the container with id f49958cd11ab8df97b263fb6e0f0b7e2394fe56a198e3e68be146ccb53e84cd3 Nov 03 16:32:20.005323 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:20.005286 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-2lqnz" event={"ID":"a305a949-c2d0-45db-ad70-7d19c6d6fbde","Type":"ContainerStarted","Data":"f49958cd11ab8df97b263fb6e0f0b7e2394fe56a198e3e68be146ccb53e84cd3"} Nov 03 16:32:21.011338 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:21.011294 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-2lqnz" event={"ID":"a305a949-c2d0-45db-ad70-7d19c6d6fbde","Type":"ContainerStarted","Data":"14c096b3bce7cf870dc4bbecb2d77c7d190dd00724cf78f6d5ca0ef5de2621c5"} Nov 03 16:32:21.011778 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:21.011386 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-2lqnz" Nov 03 16:32:21.059992 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:21.059938 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-2lqnz" podStartSLOduration=1.373076105 podStartE2EDuration="3.059923979s" totalImagesPullingTime="1.686847874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:21.058721641 +0000 UTC m=+1084.647721607" watchObservedRunningTime="2026-11-03 16:32:21.059923979 +0000 UTC m=+1084.648923946" Nov 03 16:32:23.558631 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.558593 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2"] Nov 03 16:32:23.561187 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.561166 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.578251 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.578211 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln"] Nov 03 16:32:23.580658 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.580637 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.592763 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.592733 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2"] Nov 03 16:32:23.608806 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.608763 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln"] Nov 03 16:32:23.690818 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.690780 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64bfff21-904a-40fb-829a-7f49e05f0d09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.690818 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.690832 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzqqf\" (UniqueName: \"kubernetes.io/projected/036fe6c6-378c-42f3-87cc-91315033ea59-kube-api-access-pzqqf\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.691082 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.690850 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzmdx\" (UniqueName: \"kubernetes.io/projected/64bfff21-904a-40fb-829a-7f49e05f0d09-kube-api-access-dzmdx\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.691082 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.691001 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.691082 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.691044 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.691229 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.691090 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036fe6c6-378c-42f3-87cc-91315033ea59-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.691229 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.691153 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.691229 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.691191 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.691336 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.691234 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.691336 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.691267 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.691336 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.691284 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.691336 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.691307 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.691516 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.691347 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036fe6c6-378c-42f3-87cc-91315033ea59-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.691516 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.691370 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.691516 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.691394 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.691516 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.691510 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.691730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.691540 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.691730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.691640 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036fe6c6-378c-42f3-87cc-91315033ea59-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.792340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.792302 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036fe6c6-378c-42f3-87cc-91315033ea59-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.792340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.792346 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64bfff21-904a-40fb-829a-7f49e05f0d09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.792604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.792473 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqqf\" (UniqueName: \"kubernetes.io/projected/036fe6c6-378c-42f3-87cc-91315033ea59-kube-api-access-pzqqf\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.792604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.792515 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzmdx\" (UniqueName: \"kubernetes.io/projected/64bfff21-904a-40fb-829a-7f49e05f0d09-kube-api-access-dzmdx\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.792604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.792550 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.792604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.792578 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.792848 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.792645 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036fe6c6-378c-42f3-87cc-91315033ea59-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.792848 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.792726 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.792848 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.792751 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.792848 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.792779 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.792848 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.792807 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.792848 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.792836 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.793135 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.792870 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.793135 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.792895 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036fe6c6-378c-42f3-87cc-91315033ea59-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.793135 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.792922 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.793135 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.792949 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.793135 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.792989 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.793135 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.793014 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.793426 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.793150 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64bfff21-904a-40fb-829a-7f49e05f0d09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.793426 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.793163 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036fe6c6-378c-42f3-87cc-91315033ea59-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.793426 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.793240 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.793426 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.793378 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.793426 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.793400 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.793726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.793645 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.793726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.793649 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.793726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.793684 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.793726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.793719 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.794091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.794065 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.795722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.795671 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.795722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.795677 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.795845 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.795719 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.796052 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.796031 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036fe6c6-378c-42f3-87cc-91315033ea59-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.817843 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.817768 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036fe6c6-378c-42f3-87cc-91315033ea59-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.817990 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.817910 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzmdx\" (UniqueName: \"kubernetes.io/projected/64bfff21-904a-40fb-829a-7f49e05f0d09-kube-api-access-dzmdx\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.817990 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.817922 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-x2kq2\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.818100 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.818010 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzqqf\" (UniqueName: \"kubernetes.io/projected/036fe6c6-378c-42f3-87cc-91315033ea59-kube-api-access-pzqqf\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-7zkln\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:23.874086 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.874051 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:23.896272 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:23.896244 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:24.211119 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:24.211085 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2"] Nov 03 16:32:24.213666 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:24.213643 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln"] Nov 03 16:32:24.214060 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:32:24.214032 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64bfff21_904a_40fb_829a_7f49e05f0d09.slice/crio-07cce8c7abcc2bcc98683d2978ee4cba142b0dfb332d6ffadf79de05eb8e9d86 WatchSource:0}: Error finding container 07cce8c7abcc2bcc98683d2978ee4cba142b0dfb332d6ffadf79de05eb8e9d86: Status 404 returned error can't find the container with id 07cce8c7abcc2bcc98683d2978ee4cba142b0dfb332d6ffadf79de05eb8e9d86 Nov 03 16:32:24.214989 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:32:24.214967 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036fe6c6_378c_42f3_87cc_91315033ea59.slice/crio-8a0a1809a0a147372c8975fb16f26491921e2142c214cb86d7bfec689a41e592 WatchSource:0}: Error finding container 8a0a1809a0a147372c8975fb16f26491921e2142c214cb86d7bfec689a41e592: Status 404 returned error can't find the container with id 8a0a1809a0a147372c8975fb16f26491921e2142c214cb86d7bfec689a41e592 Nov 03 16:32:24.215989 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:24.215951 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:32:24.216091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:24.216028 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:32:24.216091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:24.216067 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:32:24.217134 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:24.217110 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:32:24.217207 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:24.217165 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:32:24.217207 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:24.217193 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:32:25.042380 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:25.042342 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" event={"ID":"64bfff21-904a-40fb-829a-7f49e05f0d09","Type":"ContainerStarted","Data":"ed569fa9ca2a514b8f42ddc72260708f37149706674c344b1f3e2ded95895f15"} Nov 03 16:32:25.049034 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:25.048878 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" event={"ID":"64bfff21-904a-40fb-829a-7f49e05f0d09","Type":"ContainerStarted","Data":"07cce8c7abcc2bcc98683d2978ee4cba142b0dfb332d6ffadf79de05eb8e9d86"} Nov 03 16:32:25.051408 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:25.051307 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" event={"ID":"036fe6c6-378c-42f3-87cc-91315033ea59","Type":"ContainerStarted","Data":"01426ee92fff2e74418abcd33b6851b12a1c7f0bdfa8ff416a86576178d0059b"} Nov 03 16:32:25.051645 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:25.051624 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" event={"ID":"036fe6c6-378c-42f3-87cc-91315033ea59","Type":"ContainerStarted","Data":"8a0a1809a0a147372c8975fb16f26491921e2142c214cb86d7bfec689a41e592"} Nov 03 16:32:25.124148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:25.124089 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" podStartSLOduration=2.124070211 podStartE2EDuration="2.124070211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:25.110201892 +0000 UTC m=+1088.699201864" watchObservedRunningTime="2026-11-03 16:32:25.124070211 +0000 UTC m=+1088.713070181" Nov 03 16:32:25.160722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:25.160172 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" podStartSLOduration=2.160152944 podStartE2EDuration="2.160152944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:25.160055072 +0000 UTC m=+1088.749055055" watchObservedRunningTime="2026-11-03 16:32:25.160152944 +0000 UTC m=+1088.749152915" Nov 03 16:32:25.874360 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:25.874324 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:25.875985 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:25.875954 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:32:25.876110 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:25.876012 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" podUID="64bfff21-904a-40fb-829a-7f49e05f0d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 03 16:32:25.897183 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:25.897157 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:25.901883 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:25.901851 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7zkln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 03 16:32:25.902100 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:25.902069 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" podUID="036fe6c6-378c-42f3-87cc-91315033ea59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 03 16:32:26.874521 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:26.874483 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:32:26.874968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:26.874540 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" podUID="64bfff21-904a-40fb-829a-7f49e05f0d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 03 16:32:26.896539 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:26.896514 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7zkln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 03 16:32:26.896737 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:26.896716 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" podUID="036fe6c6-378c-42f3-87cc-91315033ea59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 03 16:32:27.875399 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:27.875357 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:32:27.875802 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:27.875414 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" podUID="64bfff21-904a-40fb-829a-7f49e05f0d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 03 16:32:27.897326 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:27.897295 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7zkln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 03 16:32:27.897454 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:27.897340 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" podUID="036fe6c6-378c-42f3-87cc-91315033ea59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 03 16:32:28.874549 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:28.874511 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:32:28.874747 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:28.874571 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" podUID="64bfff21-904a-40fb-829a-7f49e05f0d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 03 16:32:28.897408 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:28.897383 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7zkln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 03 16:32:28.897786 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:28.897431 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" podUID="036fe6c6-378c-42f3-87cc-91315033ea59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 03 16:32:29.875005 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:29.874964 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body= Nov 03 16:32:29.875179 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:29.875025 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" podUID="64bfff21-904a-40fb-829a-7f49e05f0d09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" Nov 03 16:32:29.896665 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:29.896632 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7zkln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Nov 03 16:32:29.896815 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:29.896726 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" podUID="036fe6c6-378c-42f3-87cc-91315033ea59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Nov 03 16:32:30.174277 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:32:30.174188 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:32:30.879112 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:30.879079 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:30.879412 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:30.879383 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:30.880201 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:30.880183 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:32:30.900799 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:30.900766 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:30.901059 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:30.901041 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:30.901823 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:30.901800 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 16:32:32.017762 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:32.017734 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-2lqnz" Nov 03 16:32:32.306603 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:32.306527 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d958cf575-8jkmf"] Nov 03 16:32:32.309309 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:32.309292 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d958cf575-8jkmf" Nov 03 16:32:32.338729 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:32.338671 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d958cf575-8jkmf"] Nov 03 16:32:32.410753 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:32.410690 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1ba5407-2f49-4ac7-9b41-157697ab2306-metrics-certs\") pod \"ai-gateway-controller-7d958cf575-8jkmf\" (UID: \"d1ba5407-2f49-4ac7-9b41-157697ab2306\") " pod="opendatahub/ai-gateway-controller-7d958cf575-8jkmf" Nov 03 16:32:32.410935 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:32.410838 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlhvq\" (UniqueName: \"kubernetes.io/projected/d1ba5407-2f49-4ac7-9b41-157697ab2306-kube-api-access-vlhvq\") pod \"ai-gateway-controller-7d958cf575-8jkmf\" (UID: \"d1ba5407-2f49-4ac7-9b41-157697ab2306\") " pod="opendatahub/ai-gateway-controller-7d958cf575-8jkmf" Nov 03 16:32:32.511801 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:32.511762 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlhvq\" (UniqueName: \"kubernetes.io/projected/d1ba5407-2f49-4ac7-9b41-157697ab2306-kube-api-access-vlhvq\") pod \"ai-gateway-controller-7d958cf575-8jkmf\" (UID: \"d1ba5407-2f49-4ac7-9b41-157697ab2306\") " pod="opendatahub/ai-gateway-controller-7d958cf575-8jkmf" Nov 03 16:32:32.511977 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:32.511830 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1ba5407-2f49-4ac7-9b41-157697ab2306-metrics-certs\") pod \"ai-gateway-controller-7d958cf575-8jkmf\" (UID: \"d1ba5407-2f49-4ac7-9b41-157697ab2306\") " pod="opendatahub/ai-gateway-controller-7d958cf575-8jkmf" Nov 03 16:32:32.514363 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:32.514342 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1ba5407-2f49-4ac7-9b41-157697ab2306-metrics-certs\") pod \"ai-gateway-controller-7d958cf575-8jkmf\" (UID: \"d1ba5407-2f49-4ac7-9b41-157697ab2306\") " pod="opendatahub/ai-gateway-controller-7d958cf575-8jkmf" Nov 03 16:32:32.537958 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:32.537928 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlhvq\" (UniqueName: \"kubernetes.io/projected/d1ba5407-2f49-4ac7-9b41-157697ab2306-kube-api-access-vlhvq\") pod \"ai-gateway-controller-7d958cf575-8jkmf\" (UID: \"d1ba5407-2f49-4ac7-9b41-157697ab2306\") " pod="opendatahub/ai-gateway-controller-7d958cf575-8jkmf" Nov 03 16:32:32.620829 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:32.620758 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d958cf575-8jkmf" Nov 03 16:32:32.961981 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:32.961888 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d958cf575-8jkmf"] Nov 03 16:32:32.964894 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:32:32.964872 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ba5407_2f49_4ac7_9b41_157697ab2306.slice/crio-f21da0f316ab75048dea63adb4da91f137dfa3eda026aa4b278e50eca551b36b WatchSource:0}: Error finding container f21da0f316ab75048dea63adb4da91f137dfa3eda026aa4b278e50eca551b36b: Status 404 returned error can't find the container with id f21da0f316ab75048dea63adb4da91f137dfa3eda026aa4b278e50eca551b36b Nov 03 16:32:33.090227 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:33.090192 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d958cf575-8jkmf" event={"ID":"d1ba5407-2f49-4ac7-9b41-157697ab2306","Type":"ContainerStarted","Data":"f21da0f316ab75048dea63adb4da91f137dfa3eda026aa4b278e50eca551b36b"} Nov 03 16:32:34.095925 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:34.095893 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d958cf575-8jkmf" event={"ID":"d1ba5407-2f49-4ac7-9b41-157697ab2306","Type":"ContainerStarted","Data":"7c7a6fcdfc4756a9ade09a52da22683380208c6d1e2085d37a277f11af1419e1"} Nov 03 16:32:34.096324 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:34.096033 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d958cf575-8jkmf" Nov 03 16:32:34.141643 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:34.141588 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d958cf575-8jkmf" podStartSLOduration=1.806788485 podStartE2EDuration="2.141573552s" totalImagesPullingTime="334.785067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:34.140158285 +0000 UTC m=+1097.729158254" watchObservedRunningTime="2026-11-03 16:32:34.141573552 +0000 UTC m=+1097.730573521" Nov 03 16:32:45.102904 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:45.102868 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d958cf575-8jkmf" Nov 03 16:32:45.180865 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:45.180836 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-2lqnz"] Nov 03 16:32:45.181063 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:45.181038 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-2lqnz" podUID="a305a949-c2d0-45db-ad70-7d19c6d6fbde" containerName="manager" containerID="cri-o://14c096b3bce7cf870dc4bbecb2d77c7d190dd00724cf78f6d5ca0ef5de2621c5" gracePeriod=30 Nov 03 16:32:45.433144 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:45.433120 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-2lqnz" Nov 03 16:32:45.529007 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:45.528967 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a305a949-c2d0-45db-ad70-7d19c6d6fbde-kube-api-access-mpht2\" (UniqueName: \"kubernetes.io/projected/a305a949-c2d0-45db-ad70-7d19c6d6fbde-kube-api-access-mpht2\") pod \"a305a949-c2d0-45db-ad70-7d19c6d6fbde\" (UID: \"a305a949-c2d0-45db-ad70-7d19c6d6fbde\") " Nov 03 16:32:45.529007 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:45.529013 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a305a949-c2d0-45db-ad70-7d19c6d6fbde-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a305a949-c2d0-45db-ad70-7d19c6d6fbde-metrics-certs\") pod \"a305a949-c2d0-45db-ad70-7d19c6d6fbde\" (UID: \"a305a949-c2d0-45db-ad70-7d19c6d6fbde\") " Nov 03 16:32:45.531247 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:45.531220 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a305a949-c2d0-45db-ad70-7d19c6d6fbde-kube-api-access-mpht2" pod "a305a949-c2d0-45db-ad70-7d19c6d6fbde" (UID: "a305a949-c2d0-45db-ad70-7d19c6d6fbde"). InnerVolumeSpecName "kube-api-access-mpht2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:32:45.531369 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:45.531270 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a305a949-c2d0-45db-ad70-7d19c6d6fbde-metrics-certs" pod "a305a949-c2d0-45db-ad70-7d19c6d6fbde" (UID: "a305a949-c2d0-45db-ad70-7d19c6d6fbde"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:32:45.630486 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:45.630397 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpht2\" (UniqueName: \"kubernetes.io/projected/a305a949-c2d0-45db-ad70-7d19c6d6fbde-kube-api-access-mpht2\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:32:45.630486 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:45.630436 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a305a949-c2d0-45db-ad70-7d19c6d6fbde-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:32:46.150162 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:46.150125 4947 generic.go:356] "Generic (PLEG): container finished" podID="a305a949-c2d0-45db-ad70-7d19c6d6fbde" containerID="14c096b3bce7cf870dc4bbecb2d77c7d190dd00724cf78f6d5ca0ef5de2621c5" exitCode=0 Nov 03 16:32:46.150629 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:46.150177 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-2lqnz" Nov 03 16:32:46.150629 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:46.150193 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-2lqnz" event={"ID":"a305a949-c2d0-45db-ad70-7d19c6d6fbde","Type":"ContainerDied","Data":"14c096b3bce7cf870dc4bbecb2d77c7d190dd00724cf78f6d5ca0ef5de2621c5"} Nov 03 16:32:46.150629 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:46.150245 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-2lqnz" event={"ID":"a305a949-c2d0-45db-ad70-7d19c6d6fbde","Type":"ContainerDied","Data":"f49958cd11ab8df97b263fb6e0f0b7e2394fe56a198e3e68be146ccb53e84cd3"} Nov 03 16:32:46.150629 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:46.150274 4947 scope.go:122] "RemoveContainer" containerID="14c096b3bce7cf870dc4bbecb2d77c7d190dd00724cf78f6d5ca0ef5de2621c5" Nov 03 16:32:46.159148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:46.159126 4947 scope.go:122] "RemoveContainer" containerID="14c096b3bce7cf870dc4bbecb2d77c7d190dd00724cf78f6d5ca0ef5de2621c5" Nov 03 16:32:46.159400 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:32:46.159381 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14c096b3bce7cf870dc4bbecb2d77c7d190dd00724cf78f6d5ca0ef5de2621c5\": container with ID starting with 14c096b3bce7cf870dc4bbecb2d77c7d190dd00724cf78f6d5ca0ef5de2621c5 not found: ID does not exist" containerID="14c096b3bce7cf870dc4bbecb2d77c7d190dd00724cf78f6d5ca0ef5de2621c5" Nov 03 16:32:46.159458 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:46.159410 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14c096b3bce7cf870dc4bbecb2d77c7d190dd00724cf78f6d5ca0ef5de2621c5"} err="failed to get container status \"14c096b3bce7cf870dc4bbecb2d77c7d190dd00724cf78f6d5ca0ef5de2621c5\": rpc error: code = NotFound desc = could not find container \"14c096b3bce7cf870dc4bbecb2d77c7d190dd00724cf78f6d5ca0ef5de2621c5\": container with ID starting with 14c096b3bce7cf870dc4bbecb2d77c7d190dd00724cf78f6d5ca0ef5de2621c5 not found: ID does not exist" Nov 03 16:32:46.204872 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:46.204841 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-2lqnz"] Nov 03 16:32:46.218940 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:46.218906 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-2lqnz"] Nov 03 16:32:46.568105 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:46.568071 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a305a949-c2d0-45db-ad70-7d19c6d6fbde" path="/var/lib/kubelet/pods/a305a949-c2d0-45db-ad70-7d19c6d6fbde/volumes" Nov 03 16:32:52.880652 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:52.880621 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj"] Nov 03 16:32:52.881116 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:52.881098 4947 cpu_manager.go:414] "containerMap: removing container" podUID="a305a949-c2d0-45db-ad70-7d19c6d6fbde" containerName="manager" Nov 03 16:32:52.881167 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:52.881120 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a305a949-c2d0-45db-ad70-7d19c6d6fbde" containerName="manager" Nov 03 16:32:52.881224 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:52.881214 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="a305a949-c2d0-45db-ad70-7d19c6d6fbde" containerName="manager" Nov 03 16:32:52.884899 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:52.884871 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:52.917275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:52.917240 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-znvz5\"" Nov 03 16:32:52.917275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:52.917252 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 16:32:52.917734 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:52.917715 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 16:32:52.919850 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:52.919826 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj"] Nov 03 16:32:52.920416 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:52.920400 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 03 16:32:52.994235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:52.994198 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ffcef26-eed6-4a12-86d6-a057a71db539-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj\" (UID: \"3ffcef26-eed6-4a12-86d6-a057a71db539\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:52.994235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:52.994234 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ffcef26-eed6-4a12-86d6-a057a71db539-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj\" (UID: \"3ffcef26-eed6-4a12-86d6-a057a71db539\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:52.994440 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:52.994267 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ffcef26-eed6-4a12-86d6-a057a71db539-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj\" (UID: \"3ffcef26-eed6-4a12-86d6-a057a71db539\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:52.994440 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:52.994403 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ffcef26-eed6-4a12-86d6-a057a71db539-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj\" (UID: \"3ffcef26-eed6-4a12-86d6-a057a71db539\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:52.994440 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:52.994433 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6kvq\" (UniqueName: \"kubernetes.io/projected/3ffcef26-eed6-4a12-86d6-a057a71db539-kube-api-access-t6kvq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj\" (UID: \"3ffcef26-eed6-4a12-86d6-a057a71db539\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:52.994563 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:52.994470 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ffcef26-eed6-4a12-86d6-a057a71db539-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj\" (UID: \"3ffcef26-eed6-4a12-86d6-a057a71db539\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:53.095306 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:53.095274 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ffcef26-eed6-4a12-86d6-a057a71db539-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj\" (UID: \"3ffcef26-eed6-4a12-86d6-a057a71db539\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:53.095479 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:53.095315 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ffcef26-eed6-4a12-86d6-a057a71db539-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj\" (UID: \"3ffcef26-eed6-4a12-86d6-a057a71db539\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:53.095479 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:53.095333 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6kvq\" (UniqueName: \"kubernetes.io/projected/3ffcef26-eed6-4a12-86d6-a057a71db539-kube-api-access-t6kvq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj\" (UID: \"3ffcef26-eed6-4a12-86d6-a057a71db539\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:53.095479 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:53.095456 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ffcef26-eed6-4a12-86d6-a057a71db539-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj\" (UID: \"3ffcef26-eed6-4a12-86d6-a057a71db539\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:53.095667 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:53.095580 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ffcef26-eed6-4a12-86d6-a057a71db539-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj\" (UID: \"3ffcef26-eed6-4a12-86d6-a057a71db539\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:53.095667 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:53.095607 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ffcef26-eed6-4a12-86d6-a057a71db539-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj\" (UID: \"3ffcef26-eed6-4a12-86d6-a057a71db539\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:53.095667 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:53.095655 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ffcef26-eed6-4a12-86d6-a057a71db539-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj\" (UID: \"3ffcef26-eed6-4a12-86d6-a057a71db539\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:53.095840 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:53.095773 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ffcef26-eed6-4a12-86d6-a057a71db539-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj\" (UID: \"3ffcef26-eed6-4a12-86d6-a057a71db539\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:53.095880 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:53.095838 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ffcef26-eed6-4a12-86d6-a057a71db539-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj\" (UID: \"3ffcef26-eed6-4a12-86d6-a057a71db539\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:53.098046 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:53.098020 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ffcef26-eed6-4a12-86d6-a057a71db539-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj\" (UID: \"3ffcef26-eed6-4a12-86d6-a057a71db539\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:53.098165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:53.098144 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ffcef26-eed6-4a12-86d6-a057a71db539-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj\" (UID: \"3ffcef26-eed6-4a12-86d6-a057a71db539\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:53.120541 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:53.120508 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6kvq\" (UniqueName: \"kubernetes.io/projected/3ffcef26-eed6-4a12-86d6-a057a71db539-kube-api-access-t6kvq\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj\" (UID: \"3ffcef26-eed6-4a12-86d6-a057a71db539\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:53.197662 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:53.197583 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:53.513035 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:53.513012 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj"] Nov 03 16:32:53.514645 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:32:53.514620 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffcef26_eed6_4a12_86d6_a057a71db539.slice/crio-eae4d66a8effeb91c9f55e21a2403825496a5ed0f4b3ad1f901482a71c759338 WatchSource:0}: Error finding container eae4d66a8effeb91c9f55e21a2403825496a5ed0f4b3ad1f901482a71c759338: Status 404 returned error can't find the container with id eae4d66a8effeb91c9f55e21a2403825496a5ed0f4b3ad1f901482a71c759338 Nov 03 16:32:53.891423 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:53.891341 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:32:54.197342 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:54.197206 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" event={"ID":"3ffcef26-eed6-4a12-86d6-a057a71db539","Type":"ContainerStarted","Data":"eae4d66a8effeb91c9f55e21a2403825496a5ed0f4b3ad1f901482a71c759338"} Nov 03 16:32:55.204201 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:55.204162 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" event={"ID":"3ffcef26-eed6-4a12-86d6-a057a71db539","Type":"ContainerStarted","Data":"9cf0d7cf5f724fa93353751899cc90a6363cdafe55a7c0820a0dbd4e37e8ae29"} Nov 03 16:32:55.251242 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:55.251181 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" podStartSLOduration=1.650135037 podStartE2EDuration="3.251162189s" totalImagesPullingTime="1.601027152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:32:55.249052108 +0000 UTC m=+1118.838052079" watchObservedRunningTime="2026-11-03 16:32:55.251162189 +0000 UTC m=+1118.840162159" Nov 03 16:32:56.198619 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:56.198580 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:32:57.910889 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:57.910804 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:32:58.227170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.227074 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7"] Nov 03 16:32:58.231519 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.231492 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.267792 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.267754 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 16:32:58.270402 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.270367 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7"] Nov 03 16:32:58.368210 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.368162 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eac7cf8-9b38-40e6-8864-09f26a2ff834-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zxcg7\" (UID: \"2eac7cf8-9b38-40e6-8864-09f26a2ff834\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.368511 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.368488 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg5v6\" (UniqueName: \"kubernetes.io/projected/2eac7cf8-9b38-40e6-8864-09f26a2ff834-kube-api-access-vg5v6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zxcg7\" (UID: \"2eac7cf8-9b38-40e6-8864-09f26a2ff834\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.368645 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.368629 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eac7cf8-9b38-40e6-8864-09f26a2ff834-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zxcg7\" (UID: \"2eac7cf8-9b38-40e6-8864-09f26a2ff834\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.368798 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.368781 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eac7cf8-9b38-40e6-8864-09f26a2ff834-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zxcg7\" (UID: \"2eac7cf8-9b38-40e6-8864-09f26a2ff834\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.368932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.368919 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eac7cf8-9b38-40e6-8864-09f26a2ff834-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zxcg7\" (UID: \"2eac7cf8-9b38-40e6-8864-09f26a2ff834\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.369074 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.369060 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eac7cf8-9b38-40e6-8864-09f26a2ff834-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zxcg7\" (UID: \"2eac7cf8-9b38-40e6-8864-09f26a2ff834\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.472233 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.472190 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eac7cf8-9b38-40e6-8864-09f26a2ff834-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zxcg7\" (UID: \"2eac7cf8-9b38-40e6-8864-09f26a2ff834\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.472962 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.472938 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg5v6\" (UniqueName: \"kubernetes.io/projected/2eac7cf8-9b38-40e6-8864-09f26a2ff834-kube-api-access-vg5v6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zxcg7\" (UID: \"2eac7cf8-9b38-40e6-8864-09f26a2ff834\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.473507 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.473484 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eac7cf8-9b38-40e6-8864-09f26a2ff834-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zxcg7\" (UID: \"2eac7cf8-9b38-40e6-8864-09f26a2ff834\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.474149 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.474129 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eac7cf8-9b38-40e6-8864-09f26a2ff834-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zxcg7\" (UID: \"2eac7cf8-9b38-40e6-8864-09f26a2ff834\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.474629 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.474611 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eac7cf8-9b38-40e6-8864-09f26a2ff834-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zxcg7\" (UID: \"2eac7cf8-9b38-40e6-8864-09f26a2ff834\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.475135 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.475117 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eac7cf8-9b38-40e6-8864-09f26a2ff834-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zxcg7\" (UID: \"2eac7cf8-9b38-40e6-8864-09f26a2ff834\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.475631 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.475031 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eac7cf8-9b38-40e6-8864-09f26a2ff834-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zxcg7\" (UID: \"2eac7cf8-9b38-40e6-8864-09f26a2ff834\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.475791 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.474523 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eac7cf8-9b38-40e6-8864-09f26a2ff834-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zxcg7\" (UID: \"2eac7cf8-9b38-40e6-8864-09f26a2ff834\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.475926 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.472857 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eac7cf8-9b38-40e6-8864-09f26a2ff834-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zxcg7\" (UID: \"2eac7cf8-9b38-40e6-8864-09f26a2ff834\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.485072 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.484872 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eac7cf8-9b38-40e6-8864-09f26a2ff834-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zxcg7\" (UID: \"2eac7cf8-9b38-40e6-8864-09f26a2ff834\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.487725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.487650 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eac7cf8-9b38-40e6-8864-09f26a2ff834-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zxcg7\" (UID: \"2eac7cf8-9b38-40e6-8864-09f26a2ff834\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.520431 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.520394 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg5v6\" (UniqueName: \"kubernetes.io/projected/2eac7cf8-9b38-40e6-8864-09f26a2ff834-kube-api-access-vg5v6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-zxcg7\" (UID: \"2eac7cf8-9b38-40e6-8864-09f26a2ff834\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.551193 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.550893 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:32:58.940245 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:58.940217 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7"] Nov 03 16:32:58.942798 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:32:58.942756 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eac7cf8_9b38_40e6_8864_09f26a2ff834.slice/crio-694708816e69576e15e8443bbc81ac30d7d869efa55b016b1aa4821c840759a9 WatchSource:0}: Error finding container 694708816e69576e15e8443bbc81ac30d7d869efa55b016b1aa4821c840759a9: Status 404 returned error can't find the container with id 694708816e69576e15e8443bbc81ac30d7d869efa55b016b1aa4821c840759a9 Nov 03 16:32:59.227461 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:32:59.227427 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" event={"ID":"2eac7cf8-9b38-40e6-8864-09f26a2ff834","Type":"ContainerStarted","Data":"694708816e69576e15e8443bbc81ac30d7d869efa55b016b1aa4821c840759a9"} Nov 03 16:33:00.129768 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:33:00.129728 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:33:00.235177 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:00.235137 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" event={"ID":"2eac7cf8-9b38-40e6-8864-09f26a2ff834","Type":"ContainerStarted","Data":"727bc4d441d41390c54ccf75e8718e421effb066662c149db2173f15e17dc84d"} Nov 03 16:33:00.284772 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:00.284658 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" podStartSLOduration=2.025736316 podStartE2EDuration="2.284638232s" totalImagesPullingTime="258.901916ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:00.282199379 +0000 UTC m=+1123.871199348" watchObservedRunningTime="2026-11-03 16:33:00.284638232 +0000 UTC m=+1123.873638201" Nov 03 16:33:00.551562 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:00.551523 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:33:02.466933 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:02.466896 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:33:02.862170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:02.861358 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4"] Nov 03 16:33:02.867327 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:02.867296 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:02.884448 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:02.884421 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 16:33:02.900150 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:02.900116 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4"] Nov 03 16:33:02.936377 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:02.936334 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a94473e-1c2c-470e-92a8-1f4ff82791f2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4\" (UID: \"6a94473e-1c2c-470e-92a8-1f4ff82791f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:02.936585 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:02.936468 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a94473e-1c2c-470e-92a8-1f4ff82791f2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4\" (UID: \"6a94473e-1c2c-470e-92a8-1f4ff82791f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:02.936585 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:02.936531 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a94473e-1c2c-470e-92a8-1f4ff82791f2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4\" (UID: \"6a94473e-1c2c-470e-92a8-1f4ff82791f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:02.936585 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:02.936581 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a94473e-1c2c-470e-92a8-1f4ff82791f2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4\" (UID: \"6a94473e-1c2c-470e-92a8-1f4ff82791f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:02.936797 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:02.936612 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdmg9\" (UniqueName: \"kubernetes.io/projected/6a94473e-1c2c-470e-92a8-1f4ff82791f2-kube-api-access-rdmg9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4\" (UID: \"6a94473e-1c2c-470e-92a8-1f4ff82791f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:02.936797 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:02.936763 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a94473e-1c2c-470e-92a8-1f4ff82791f2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4\" (UID: \"6a94473e-1c2c-470e-92a8-1f4ff82791f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:03.038018 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:03.037981 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a94473e-1c2c-470e-92a8-1f4ff82791f2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4\" (UID: \"6a94473e-1c2c-470e-92a8-1f4ff82791f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:03.038224 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:03.038061 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a94473e-1c2c-470e-92a8-1f4ff82791f2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4\" (UID: \"6a94473e-1c2c-470e-92a8-1f4ff82791f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:03.038224 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:03.038088 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a94473e-1c2c-470e-92a8-1f4ff82791f2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4\" (UID: \"6a94473e-1c2c-470e-92a8-1f4ff82791f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:03.038224 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:03.038139 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a94473e-1c2c-470e-92a8-1f4ff82791f2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4\" (UID: \"6a94473e-1c2c-470e-92a8-1f4ff82791f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:03.038224 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:03.038183 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a94473e-1c2c-470e-92a8-1f4ff82791f2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4\" (UID: \"6a94473e-1c2c-470e-92a8-1f4ff82791f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:03.038224 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:03.038212 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdmg9\" (UniqueName: \"kubernetes.io/projected/6a94473e-1c2c-470e-92a8-1f4ff82791f2-kube-api-access-rdmg9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4\" (UID: \"6a94473e-1c2c-470e-92a8-1f4ff82791f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:03.038558 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:03.038530 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a94473e-1c2c-470e-92a8-1f4ff82791f2-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4\" (UID: \"6a94473e-1c2c-470e-92a8-1f4ff82791f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:03.038823 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:03.038804 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a94473e-1c2c-470e-92a8-1f4ff82791f2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4\" (UID: \"6a94473e-1c2c-470e-92a8-1f4ff82791f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:03.038919 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:03.038890 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a94473e-1c2c-470e-92a8-1f4ff82791f2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4\" (UID: \"6a94473e-1c2c-470e-92a8-1f4ff82791f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:03.041442 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:03.041410 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a94473e-1c2c-470e-92a8-1f4ff82791f2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4\" (UID: \"6a94473e-1c2c-470e-92a8-1f4ff82791f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:03.042088 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:03.042066 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a94473e-1c2c-470e-92a8-1f4ff82791f2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4\" (UID: \"6a94473e-1c2c-470e-92a8-1f4ff82791f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:03.066546 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:03.066510 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdmg9\" (UniqueName: \"kubernetes.io/projected/6a94473e-1c2c-470e-92a8-1f4ff82791f2-kube-api-access-rdmg9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4\" (UID: \"6a94473e-1c2c-470e-92a8-1f4ff82791f2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:03.186621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:03.186524 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:03.197892 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:03.197866 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:33:03.215737 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:03.215671 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:33:03.285611 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:03.285578 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj" Nov 03 16:33:03.540724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:03.540642 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4"] Nov 03 16:33:04.274729 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:04.274554 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" event={"ID":"6a94473e-1c2c-470e-92a8-1f4ff82791f2","Type":"ContainerStarted","Data":"f5b9c2f0c80810fb6d147023754165b53a036bf2e8094c67ba7c25d1e6cb8f27"} Nov 03 16:33:04.274729 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:04.274616 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" event={"ID":"6a94473e-1c2c-470e-92a8-1f4ff82791f2","Type":"ContainerStarted","Data":"c6295c1c1a79d1657177fd805aa4c87eb2d0461651d2dda03944cad0980575e3"} Nov 03 16:33:04.322488 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:04.322426 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" podStartSLOduration=1.921503506 podStartE2EDuration="2.322406557s" totalImagesPullingTime="400.903051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:04.321202971 +0000 UTC m=+1127.910202941" watchObservedRunningTime="2026-11-03 16:33:04.322406557 +0000 UTC m=+1127.911406526" Nov 03 16:33:05.187595 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:05.187554 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:07.548597 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:07.548560 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:33:08.551867 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:08.551826 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:33:08.571030 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:08.571002 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:33:09.338726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:09.328019 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-zxcg7" Nov 03 16:33:10.791321 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.791282 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj"] Nov 03 16:33:10.795947 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.795922 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:10.822176 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.822143 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj"] Nov 03 16:33:10.833357 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.833325 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 16:33:10.889809 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.889769 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7be4c6-a940-4bb9-a6f6-076d90494556-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj\" (UID: \"6f7be4c6-a940-4bb9-a6f6-076d90494556\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:10.889993 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.889820 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7be4c6-a940-4bb9-a6f6-076d90494556-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj\" (UID: \"6f7be4c6-a940-4bb9-a6f6-076d90494556\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:10.889993 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.889907 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7be4c6-a940-4bb9-a6f6-076d90494556-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj\" (UID: \"6f7be4c6-a940-4bb9-a6f6-076d90494556\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:10.889993 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.889933 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7be4c6-a940-4bb9-a6f6-076d90494556-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj\" (UID: \"6f7be4c6-a940-4bb9-a6f6-076d90494556\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:10.890094 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.890059 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7be4c6-a940-4bb9-a6f6-076d90494556-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj\" (UID: \"6f7be4c6-a940-4bb9-a6f6-076d90494556\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:10.890127 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.890106 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lttrd\" (UniqueName: \"kubernetes.io/projected/6f7be4c6-a940-4bb9-a6f6-076d90494556-kube-api-access-lttrd\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj\" (UID: \"6f7be4c6-a940-4bb9-a6f6-076d90494556\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:10.993014 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.992976 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7be4c6-a940-4bb9-a6f6-076d90494556-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj\" (UID: \"6f7be4c6-a940-4bb9-a6f6-076d90494556\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:10.993736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.993711 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7be4c6-a940-4bb9-a6f6-076d90494556-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj\" (UID: \"6f7be4c6-a940-4bb9-a6f6-076d90494556\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:10.993925 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.993910 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7be4c6-a940-4bb9-a6f6-076d90494556-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj\" (UID: \"6f7be4c6-a940-4bb9-a6f6-076d90494556\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:10.994152 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.994022 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7be4c6-a940-4bb9-a6f6-076d90494556-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj\" (UID: \"6f7be4c6-a940-4bb9-a6f6-076d90494556\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:10.994152 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.994124 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7be4c6-a940-4bb9-a6f6-076d90494556-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj\" (UID: \"6f7be4c6-a940-4bb9-a6f6-076d90494556\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:10.994345 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.994175 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lttrd\" (UniqueName: \"kubernetes.io/projected/6f7be4c6-a940-4bb9-a6f6-076d90494556-kube-api-access-lttrd\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj\" (UID: \"6f7be4c6-a940-4bb9-a6f6-076d90494556\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:10.994345 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.994221 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7be4c6-a940-4bb9-a6f6-076d90494556-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj\" (UID: \"6f7be4c6-a940-4bb9-a6f6-076d90494556\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:10.994841 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.994820 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7be4c6-a940-4bb9-a6f6-076d90494556-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj\" (UID: \"6f7be4c6-a940-4bb9-a6f6-076d90494556\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:10.995258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.995142 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7be4c6-a940-4bb9-a6f6-076d90494556-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj\" (UID: \"6f7be4c6-a940-4bb9-a6f6-076d90494556\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:10.996299 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.996278 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7be4c6-a940-4bb9-a6f6-076d90494556-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj\" (UID: \"6f7be4c6-a940-4bb9-a6f6-076d90494556\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:10.998236 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:10.998213 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7be4c6-a940-4bb9-a6f6-076d90494556-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj\" (UID: \"6f7be4c6-a940-4bb9-a6f6-076d90494556\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:11.019253 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:11.019221 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lttrd\" (UniqueName: \"kubernetes.io/projected/6f7be4c6-a940-4bb9-a6f6-076d90494556-kube-api-access-lttrd\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj\" (UID: \"6f7be4c6-a940-4bb9-a6f6-076d90494556\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:11.113626 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:11.113539 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:11.417433 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:11.417409 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj"] Nov 03 16:33:11.418454 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:33:11.418421 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7be4c6_a940_4bb9_a6f6_076d90494556.slice/crio-0627f6041ad2dfa758052a35f7fc94cb184418fa5964247646e07c5334215b22 WatchSource:0}: Error finding container 0627f6041ad2dfa758052a35f7fc94cb184418fa5964247646e07c5334215b22: Status 404 returned error can't find the container with id 0627f6041ad2dfa758052a35f7fc94cb184418fa5964247646e07c5334215b22 Nov 03 16:33:12.319679 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:12.319643 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" event={"ID":"6f7be4c6-a940-4bb9-a6f6-076d90494556","Type":"ContainerStarted","Data":"0cd4d29038c8d9c1b3e394986598b7211d0624621a48b0aff703547cf6e111ca"} Nov 03 16:33:12.320123 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:12.319753 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" event={"ID":"6f7be4c6-a940-4bb9-a6f6-076d90494556","Type":"ContainerStarted","Data":"0627f6041ad2dfa758052a35f7fc94cb184418fa5964247646e07c5334215b22"} Nov 03 16:33:12.367019 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:12.366964 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" podStartSLOduration=2.125195833 podStartE2EDuration="2.366949342s" totalImagesPullingTime="241.753509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:33:12.36525387 +0000 UTC m=+1135.954253839" watchObservedRunningTime="2026-11-03 16:33:12.366949342 +0000 UTC m=+1135.955949310" Nov 03 16:33:12.541763 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:12.541726 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:33:13.113935 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:13.113892 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:13.187255 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:13.187161 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:13.207042 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:13.207008 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:13.343743 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:13.343683 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4" Nov 03 16:33:18.913720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:18.912856 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rp689 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:33:18.913720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:18.913009 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" podUID="43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:33:19.226791 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:19.225588 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:33:21.114657 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:21.114616 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:21.127820 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:21.127791 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:21.399662 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:33:21.399583 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj" Nov 03 16:33:30.130029 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:33:30.129990 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:34:00.129681 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:34:00.129646 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:34:06.326271 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.326238 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w"] Nov 03 16:34:06.329910 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.329892 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.359944 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.359906 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c9fa1268-openshift-default-dockercfg-qtfxm\"" Nov 03 16:34:06.374082 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.374054 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w"] Nov 03 16:34:06.386909 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.386882 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-token\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.387163 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.386920 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-workload-certs\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.387163 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.386947 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brj2w\" (UniqueName: \"kubernetes.io/projected/6bc977de-5e83-4494-a9aa-1214a2cd29e1-kube-api-access-brj2w\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.387163 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.387052 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-envoy\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.387163 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.387083 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-data\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.387163 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.387137 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istiod-ca-cert\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.387418 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.387171 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-podinfo\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.387418 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.387274 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-credential-socket\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.387418 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.387358 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-workload-socket\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.481082 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.481048 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2"] Nov 03 16:34:06.487320 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.486671 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.488665 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.488623 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istiod-ca-cert\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.488931 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.488910 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-podinfo\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.489109 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.489091 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-credential-socket\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.489237 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.489222 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-workload-socket\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.489386 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.489369 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-token\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.489505 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.489491 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-workload-certs\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.489629 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.489599 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brj2w\" (UniqueName: \"kubernetes.io/projected/6bc977de-5e83-4494-a9aa-1214a2cd29e1-kube-api-access-brj2w\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.489783 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.489662 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-envoy\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.489783 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.489717 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-data\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.490575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.490168 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-workload-socket\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.490575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.490237 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-data\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.490575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.490386 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-workload-certs\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.490575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.490509 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istiod-ca-cert\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.490877 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.490794 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-credential-socket\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.492983 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.492923 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-podinfo\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.494714 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.494677 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-envoy\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.504761 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.504731 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f0b922f4-openshift-default-dockercfg-sm98l\"" Nov 03 16:34:06.530606 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.530574 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brj2w\" (UniqueName: \"kubernetes.io/projected/6bc977de-5e83-4494-a9aa-1214a2cd29e1-kube-api-access-brj2w\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.532577 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.532546 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2"] Nov 03 16:34:06.536147 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.536121 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-token\") pod \"e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.590432 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.590337 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-istio-data\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.590432 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.590379 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df03b919-6262-4fd9-af88-788700865ebd-istio-token\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.590432 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.590401 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-workload-socket\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.590764 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.590558 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcz68\" (UniqueName: \"kubernetes.io/projected/df03b919-6262-4fd9-af88-788700865ebd-kube-api-access-pcz68\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.590764 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.590618 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-istio-envoy\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.590764 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.590671 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df03b919-6262-4fd9-af88-788700865ebd-istiod-ca-cert\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.590895 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.590804 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df03b919-6262-4fd9-af88-788700865ebd-istio-podinfo\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.590895 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.590838 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-credential-socket\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.590895 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.590860 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-workload-certs\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.643161 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.643122 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:06.694227 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.694190 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df03b919-6262-4fd9-af88-788700865ebd-istio-podinfo\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.694453 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.694243 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-credential-socket\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.694453 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.694281 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-workload-certs\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.694453 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.694327 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-istio-data\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.694453 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.694358 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df03b919-6262-4fd9-af88-788700865ebd-istio-token\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.694453 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.694382 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-workload-socket\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.694453 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.694415 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcz68\" (UniqueName: \"kubernetes.io/projected/df03b919-6262-4fd9-af88-788700865ebd-kube-api-access-pcz68\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.694840 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.694473 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-istio-envoy\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.694840 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.694498 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df03b919-6262-4fd9-af88-788700865ebd-istiod-ca-cert\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.694958 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.694855 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-workload-certs\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.694958 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.694877 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-istio-data\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.694958 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.694944 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-credential-socket\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.695157 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.695132 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-workload-socket\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.695309 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.695290 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df03b919-6262-4fd9-af88-788700865ebd-istiod-ca-cert\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.699610 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.699581 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-istio-envoy\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.702439 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.702411 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df03b919-6262-4fd9-af88-788700865ebd-istio-podinfo\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.721709 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.721625 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcz68\" (UniqueName: \"kubernetes.io/projected/df03b919-6262-4fd9-af88-788700865ebd-kube-api-access-pcz68\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.750739 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.748411 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df03b919-6262-4fd9-af88-788700865ebd-istio-token\") pod \"e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.819681 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.819643 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:06.973299 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.973268 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w"] Nov 03 16:34:06.987127 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.987096 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:06.987217 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.987172 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:06.987217 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:06.987201 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:07.127824 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:07.127020 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2"] Nov 03 16:34:07.131330 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:07.131290 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:07.131489 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:07.131466 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:07.131565 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:07.131518 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:07.602374 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:07.602337 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" event={"ID":"df03b919-6262-4fd9-af88-788700865ebd","Type":"ContainerStarted","Data":"f4ad9579bc3c016d75dc2032b2d60887c6cb8a76bbb54bd419bcacb95f0ed15e"} Nov 03 16:34:07.602818 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:07.602400 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" event={"ID":"df03b919-6262-4fd9-af88-788700865ebd","Type":"ContainerStarted","Data":"e9eb6db7118a69c799eef05dae3c724e2699d4b12d93794ccfa5275a707485f4"} Nov 03 16:34:07.603950 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:07.603928 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" event={"ID":"6bc977de-5e83-4494-a9aa-1214a2cd29e1","Type":"ContainerStarted","Data":"c228478adf988b477f285ce01dc6f939fd3495e61884f6c0f236bf9100ce624e"} Nov 03 16:34:07.604073 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:07.603975 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" event={"ID":"6bc977de-5e83-4494-a9aa-1214a2cd29e1","Type":"ContainerStarted","Data":"8655399a0c7e5fe47a9dd1d5d8d65101e4e4d49267c3fe25c0ce571084e0e307"} Nov 03 16:34:07.644212 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:07.644179 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:07.651574 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:07.651514 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" podStartSLOduration=1.651496562 podStartE2EDuration="1.651496562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:07.650934477 +0000 UTC m=+1191.239934446" watchObservedRunningTime="2026-11-03 16:34:07.651496562 +0000 UTC m=+1191.240496531" Nov 03 16:34:07.703768 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:07.703683 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" podStartSLOduration=1.70366828 podStartE2EDuration="1.70366828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:07.702124251 +0000 UTC m=+1191.291124219" watchObservedRunningTime="2026-11-03 16:34:07.70366828 +0000 UTC m=+1191.292668296" Nov 03 16:34:07.820402 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:07.820372 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:08.649072 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:08.649042 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:08.826592 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:08.826565 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:09.612990 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:09.612954 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:09.613314 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:09.613287 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:09.613827 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:09.613802 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:09.613982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:09.613968 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:10.827478 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:10.827440 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s"] Nov 03 16:34:10.831236 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:10.831213 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" Nov 03 16:34:10.852568 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:10.852540 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-cacca478-serving-cert\"" Nov 03 16:34:10.868125 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:10.868083 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s"] Nov 03 16:34:10.953184 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:10.953135 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-metrics-certs\") pod \"maas-api-e2e-migrate-cacca478-bcc47584-thd5s\" (UID: \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" Nov 03 16:34:10.953378 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:10.953292 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-maas-api-tls\") pod \"maas-api-e2e-migrate-cacca478-bcc47584-thd5s\" (UID: \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" Nov 03 16:34:10.953430 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:10.953377 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28s49\" (UniqueName: \"kubernetes.io/projected/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-kube-api-access-28s49\") pod \"maas-api-e2e-migrate-cacca478-bcc47584-thd5s\" (UID: \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" Nov 03 16:34:10.953479 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:10.953464 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-openshift-service-ca\") pod \"maas-api-e2e-migrate-cacca478-bcc47584-thd5s\" (UID: \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" Nov 03 16:34:11.054664 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:11.054631 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-openshift-service-ca\") pod \"maas-api-e2e-migrate-cacca478-bcc47584-thd5s\" (UID: \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" Nov 03 16:34:11.054890 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:11.054708 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-metrics-certs\") pod \"maas-api-e2e-migrate-cacca478-bcc47584-thd5s\" (UID: \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" Nov 03 16:34:11.054890 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:11.054763 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-maas-api-tls\") pod \"maas-api-e2e-migrate-cacca478-bcc47584-thd5s\" (UID: \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" Nov 03 16:34:11.054890 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:11.054789 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28s49\" (UniqueName: \"kubernetes.io/projected/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-kube-api-access-28s49\") pod \"maas-api-e2e-migrate-cacca478-bcc47584-thd5s\" (UID: \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" Nov 03 16:34:11.055428 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:11.055406 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-openshift-service-ca\") pod \"maas-api-e2e-migrate-cacca478-bcc47584-thd5s\" (UID: \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" Nov 03 16:34:11.057293 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:11.057275 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-metrics-certs\") pod \"maas-api-e2e-migrate-cacca478-bcc47584-thd5s\" (UID: \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" Nov 03 16:34:11.057431 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:11.057412 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-maas-api-tls\") pod \"maas-api-e2e-migrate-cacca478-bcc47584-thd5s\" (UID: \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" Nov 03 16:34:11.083820 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:11.083755 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28s49\" (UniqueName: \"kubernetes.io/projected/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-kube-api-access-28s49\") pod \"maas-api-e2e-migrate-cacca478-bcc47584-thd5s\" (UID: \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" Nov 03 16:34:11.159678 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:11.158899 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" Nov 03 16:34:11.493254 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:11.493225 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s"] Nov 03 16:34:11.494877 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:34:11.494852 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6bcd10_458d_45f7_bfd7_729e65ad9cf8.slice/crio-5980315c674f9be58e6089e0afd90dd58669f42d2c8a82ebfdc6447aeeb17a7b WatchSource:0}: Error finding container 5980315c674f9be58e6089e0afd90dd58669f42d2c8a82ebfdc6447aeeb17a7b: Status 404 returned error can't find the container with id 5980315c674f9be58e6089e0afd90dd58669f42d2c8a82ebfdc6447aeeb17a7b Nov 03 16:34:11.623195 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:11.623159 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" event={"ID":"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8","Type":"ContainerStarted","Data":"5980315c674f9be58e6089e0afd90dd58669f42d2c8a82ebfdc6447aeeb17a7b"} Nov 03 16:34:12.630105 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.630070 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" event={"ID":"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8","Type":"ContainerStarted","Data":"52e0e174a7cb554f5b99c121c58e184d6009abcce506ea82091604f0c2f4bd37"} Nov 03 16:34:12.630556 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.630163 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" Nov 03 16:34:12.669972 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.669939 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx"] Nov 03 16:34:12.673303 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.673286 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" Nov 03 16:34:12.690828 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.690799 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f0b922f4-serving-cert\"" Nov 03 16:34:12.708267 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.707617 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" podStartSLOduration=2.339409345 podStartE2EDuration="2.707595196s" totalImagesPullingTime="368.185851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:12.695049991 +0000 UTC m=+1196.284049960" watchObservedRunningTime="2026-11-03 16:34:12.707595196 +0000 UTC m=+1196.296595167" Nov 03 16:34:12.720161 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.720097 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx"] Nov 03 16:34:12.778206 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.778157 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-maas-api-tls\") pod \"maas-api-e2e-ait-f0b922f4-784778d58-dc9zx\" (UID: \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" Nov 03 16:34:12.778412 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.778303 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42jc4\" (UniqueName: \"kubernetes.io/projected/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-kube-api-access-42jc4\") pod \"maas-api-e2e-ait-f0b922f4-784778d58-dc9zx\" (UID: \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" Nov 03 16:34:12.778412 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.778330 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-openshift-service-ca\") pod \"maas-api-e2e-ait-f0b922f4-784778d58-dc9zx\" (UID: \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" Nov 03 16:34:12.778412 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.778361 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-metrics-certs\") pod \"maas-api-e2e-ait-f0b922f4-784778d58-dc9zx\" (UID: \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" Nov 03 16:34:12.879081 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.879040 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-metrics-certs\") pod \"maas-api-e2e-ait-f0b922f4-784778d58-dc9zx\" (UID: \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" Nov 03 16:34:12.879270 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.879146 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-maas-api-tls\") pod \"maas-api-e2e-ait-f0b922f4-784778d58-dc9zx\" (UID: \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" Nov 03 16:34:12.879270 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.879249 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42jc4\" (UniqueName: \"kubernetes.io/projected/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-kube-api-access-42jc4\") pod \"maas-api-e2e-ait-f0b922f4-784778d58-dc9zx\" (UID: \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" Nov 03 16:34:12.879384 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.879273 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-openshift-service-ca\") pod \"maas-api-e2e-ait-f0b922f4-784778d58-dc9zx\" (UID: \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" Nov 03 16:34:12.880101 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.880055 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-openshift-service-ca\") pod \"maas-api-e2e-ait-f0b922f4-784778d58-dc9zx\" (UID: \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" Nov 03 16:34:12.881961 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.881902 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-maas-api-tls\") pod \"maas-api-e2e-ait-f0b922f4-784778d58-dc9zx\" (UID: \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" Nov 03 16:34:12.882046 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.881978 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-metrics-certs\") pod \"maas-api-e2e-ait-f0b922f4-784778d58-dc9zx\" (UID: \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" Nov 03 16:34:12.905224 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.905188 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42jc4\" (UniqueName: \"kubernetes.io/projected/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-kube-api-access-42jc4\") pod \"maas-api-e2e-ait-f0b922f4-784778d58-dc9zx\" (UID: \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" Nov 03 16:34:12.987662 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:12.987622 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" Nov 03 16:34:13.113796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:13.111286 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:34:13.306482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:13.306447 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx"] Nov 03 16:34:13.307740 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:34:13.307681 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be3a9c7_cec4_46b2_87cd_09f44f31afc2.slice/crio-7657d77eaa62ba2d10fd2be68eec150fdeb121816a8e573c7c23055048bf3dc3 WatchSource:0}: Error finding container 7657d77eaa62ba2d10fd2be68eec150fdeb121816a8e573c7c23055048bf3dc3: Status 404 returned error can't find the container with id 7657d77eaa62ba2d10fd2be68eec150fdeb121816a8e573c7c23055048bf3dc3 Nov 03 16:34:13.636350 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:13.636316 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" event={"ID":"6be3a9c7-cec4-46b2-87cd-09f44f31afc2","Type":"ContainerStarted","Data":"7657d77eaa62ba2d10fd2be68eec150fdeb121816a8e573c7c23055048bf3dc3"} Nov 03 16:34:14.037448 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.028873 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p"] Nov 03 16:34:14.041781 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.041752 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p" Nov 03 16:34:14.046793 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.046765 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p"] Nov 03 16:34:14.057936 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.057913 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c9fa1268\"" Nov 03 16:34:14.058093 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.058006 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c9fa1268-dockercfg-4g2d5\"" Nov 03 16:34:14.139217 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.139177 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n9mw\" (UniqueName: \"kubernetes.io/projected/2bdcb312-0f38-400e-a242-4c3d3f426179-kube-api-access-6n9mw\") pod \"payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p\" (UID: \"2bdcb312-0f38-400e-a242-4c3d3f426179\") " pod="openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p" Nov 03 16:34:14.139375 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.139300 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bdcb312-0f38-400e-a242-4c3d3f426179-plugins-config-volume\") pod \"payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p\" (UID: \"2bdcb312-0f38-400e-a242-4c3d3f426179\") " pod="openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p" Nov 03 16:34:14.240256 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.240223 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n9mw\" (UniqueName: \"kubernetes.io/projected/2bdcb312-0f38-400e-a242-4c3d3f426179-kube-api-access-6n9mw\") pod \"payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p\" (UID: \"2bdcb312-0f38-400e-a242-4c3d3f426179\") " pod="openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p" Nov 03 16:34:14.240422 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.240279 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bdcb312-0f38-400e-a242-4c3d3f426179-plugins-config-volume\") pod \"payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p\" (UID: \"2bdcb312-0f38-400e-a242-4c3d3f426179\") " pod="openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p" Nov 03 16:34:14.240974 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.240956 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bdcb312-0f38-400e-a242-4c3d3f426179-plugins-config-volume\") pod \"payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p\" (UID: \"2bdcb312-0f38-400e-a242-4c3d3f426179\") " pod="openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p" Nov 03 16:34:14.264790 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.264760 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n9mw\" (UniqueName: \"kubernetes.io/projected/2bdcb312-0f38-400e-a242-4c3d3f426179-kube-api-access-6n9mw\") pod \"payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p\" (UID: \"2bdcb312-0f38-400e-a242-4c3d3f426179\") " pod="openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p" Nov 03 16:34:14.360760 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.359932 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p" Nov 03 16:34:14.643135 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.643046 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" event={"ID":"6be3a9c7-cec4-46b2-87cd-09f44f31afc2","Type":"ContainerStarted","Data":"bcaade0418ac96f688a63a9da29810eb0c19d2af75ac1156bb46e653de6799dc"} Nov 03 16:34:14.643596 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.643225 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" Nov 03 16:34:14.668765 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.668734 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p"] Nov 03 16:34:14.670240 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:34:14.670211 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bdcb312_0f38_400e_a242_4c3d3f426179.slice/crio-915df1932ba160fa0f031d5170a0b00d33fcde84ed9fe18df5184aed97330554 WatchSource:0}: Error finding container 915df1932ba160fa0f031d5170a0b00d33fcde84ed9fe18df5184aed97330554: Status 404 returned error can't find the container with id 915df1932ba160fa0f031d5170a0b00d33fcde84ed9fe18df5184aed97330554 Nov 03 16:34:14.694671 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.694613 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" podStartSLOduration=2.406211855 podStartE2EDuration="2.694597347s" totalImagesPullingTime="288.385492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:14.693209671 +0000 UTC m=+1198.282209641" watchObservedRunningTime="2026-11-03 16:34:14.694597347 +0000 UTC m=+1198.283597352" Nov 03 16:34:14.709505 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.709471 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5"] Nov 03 16:34:14.714439 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.714414 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" Nov 03 16:34:14.737062 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.737012 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c9fa1268-serving-cert\"" Nov 03 16:34:14.746060 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.745928 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5"] Nov 03 16:34:14.847340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.847303 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cc265f-63b8-4014-a19f-708f8c38d6a7-maas-api-tls\") pod \"maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5\" (UID: \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" Nov 03 16:34:14.847532 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.847347 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qwlz\" (UniqueName: \"kubernetes.io/projected/a8cc265f-63b8-4014-a19f-708f8c38d6a7-kube-api-access-2qwlz\") pod \"maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5\" (UID: \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" Nov 03 16:34:14.847532 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.847465 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cc265f-63b8-4014-a19f-708f8c38d6a7-metrics-certs\") pod \"maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5\" (UID: \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" Nov 03 16:34:14.847619 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.847598 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cc265f-63b8-4014-a19f-708f8c38d6a7-openshift-service-ca\") pod \"maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5\" (UID: \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" Nov 03 16:34:14.949223 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.949136 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cc265f-63b8-4014-a19f-708f8c38d6a7-openshift-service-ca\") pod \"maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5\" (UID: \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" Nov 03 16:34:14.949223 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.949202 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cc265f-63b8-4014-a19f-708f8c38d6a7-maas-api-tls\") pod \"maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5\" (UID: \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" Nov 03 16:34:14.949407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.949227 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qwlz\" (UniqueName: \"kubernetes.io/projected/a8cc265f-63b8-4014-a19f-708f8c38d6a7-kube-api-access-2qwlz\") pod \"maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5\" (UID: \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" Nov 03 16:34:14.949407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.949264 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cc265f-63b8-4014-a19f-708f8c38d6a7-metrics-certs\") pod \"maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5\" (UID: \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" Nov 03 16:34:14.949835 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.949808 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cc265f-63b8-4014-a19f-708f8c38d6a7-openshift-service-ca\") pod \"maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5\" (UID: \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" Nov 03 16:34:14.951770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.951744 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cc265f-63b8-4014-a19f-708f8c38d6a7-maas-api-tls\") pod \"maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5\" (UID: \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" Nov 03 16:34:14.951903 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.951883 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cc265f-63b8-4014-a19f-708f8c38d6a7-metrics-certs\") pod \"maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5\" (UID: \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" Nov 03 16:34:14.975639 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:14.975603 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qwlz\" (UniqueName: \"kubernetes.io/projected/a8cc265f-63b8-4014-a19f-708f8c38d6a7-kube-api-access-2qwlz\") pod \"maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5\" (UID: \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" Nov 03 16:34:15.029396 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:15.029153 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" Nov 03 16:34:15.345333 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:15.345303 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5"] Nov 03 16:34:15.347246 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:34:15.347216 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cc265f_63b8_4014_a19f_708f8c38d6a7.slice/crio-a0c36c54ffde7d69ae132e249fe2dec3ca3482c8351a56f7aee0d4a5563c5f00 WatchSource:0}: Error finding container a0c36c54ffde7d69ae132e249fe2dec3ca3482c8351a56f7aee0d4a5563c5f00: Status 404 returned error can't find the container with id a0c36c54ffde7d69ae132e249fe2dec3ca3482c8351a56f7aee0d4a5563c5f00 Nov 03 16:34:15.653434 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:15.653390 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" event={"ID":"a8cc265f-63b8-4014-a19f-708f8c38d6a7","Type":"ContainerStarted","Data":"a0c36c54ffde7d69ae132e249fe2dec3ca3482c8351a56f7aee0d4a5563c5f00"} Nov 03 16:34:15.655675 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:15.655641 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p" event={"ID":"2bdcb312-0f38-400e-a242-4c3d3f426179","Type":"ContainerStarted","Data":"915df1932ba160fa0f031d5170a0b00d33fcde84ed9fe18df5184aed97330554"} Nov 03 16:34:16.671655 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.671580 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" event={"ID":"a8cc265f-63b8-4014-a19f-708f8c38d6a7","Type":"ContainerStarted","Data":"83dbd7290bb37b6ab476149a3d85fafdcc4c61042d4b24ee35882331913501fb"} Nov 03 16:34:16.672198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.671771 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" Nov 03 16:34:16.682946 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.682917 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:34:16.687316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.687289 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:34:16.738468 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.738414 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" podStartSLOduration=2.461455275 podStartE2EDuration="2.738394429s" totalImagesPullingTime="276.939154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:16.726253319 +0000 UTC m=+1200.315253289" watchObservedRunningTime="2026-11-03 16:34:16.738394429 +0000 UTC m=+1200.327394399" Nov 03 16:34:16.739157 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.739136 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm"] Nov 03 16:34:16.743198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.743176 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:16.774068 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.773605 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b5770a0e-serving-cert\"" Nov 03 16:34:16.774253 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.774225 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm"] Nov 03 16:34:16.875829 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.875801 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x98cc\" (UniqueName: \"kubernetes.io/projected/252fe161-b368-435b-8c08-f8e2b241753a-kube-api-access-x98cc\") pod \"maas-api-e2e-ait-b5770a0e-594c557f44-fctpm\" (UID: \"252fe161-b368-435b-8c08-f8e2b241753a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:16.875939 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.875872 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/252fe161-b368-435b-8c08-f8e2b241753a-metrics-certs\") pod \"maas-api-e2e-ait-b5770a0e-594c557f44-fctpm\" (UID: \"252fe161-b368-435b-8c08-f8e2b241753a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:16.875983 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.875961 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/252fe161-b368-435b-8c08-f8e2b241753a-maas-api-tls\") pod \"maas-api-e2e-ait-b5770a0e-594c557f44-fctpm\" (UID: \"252fe161-b368-435b-8c08-f8e2b241753a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:16.876044 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.876010 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/252fe161-b368-435b-8c08-f8e2b241753a-openshift-service-ca\") pod \"maas-api-e2e-ait-b5770a0e-594c557f44-fctpm\" (UID: \"252fe161-b368-435b-8c08-f8e2b241753a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:16.976770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.976730 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/252fe161-b368-435b-8c08-f8e2b241753a-metrics-certs\") pod \"maas-api-e2e-ait-b5770a0e-594c557f44-fctpm\" (UID: \"252fe161-b368-435b-8c08-f8e2b241753a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:16.977580 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.977534 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/252fe161-b368-435b-8c08-f8e2b241753a-maas-api-tls\") pod \"maas-api-e2e-ait-b5770a0e-594c557f44-fctpm\" (UID: \"252fe161-b368-435b-8c08-f8e2b241753a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:16.977741 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.977619 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/252fe161-b368-435b-8c08-f8e2b241753a-openshift-service-ca\") pod \"maas-api-e2e-ait-b5770a0e-594c557f44-fctpm\" (UID: \"252fe161-b368-435b-8c08-f8e2b241753a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:16.977741 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.977730 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x98cc\" (UniqueName: \"kubernetes.io/projected/252fe161-b368-435b-8c08-f8e2b241753a-kube-api-access-x98cc\") pod \"maas-api-e2e-ait-b5770a0e-594c557f44-fctpm\" (UID: \"252fe161-b368-435b-8c08-f8e2b241753a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:16.978020 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:34:16.977994 4947 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-serving-cert: secret "maas-api-e2e-ait-b5770a0e-serving-cert" not found Nov 03 16:34:16.978148 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:34:16.978071 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/252fe161-b368-435b-8c08-f8e2b241753a-maas-api-tls podName:252fe161-b368-435b-8c08-f8e2b241753a nodeName:}" failed. No retries permitted until 2026-11-03 16:34:17.478049063 +0000 UTC m=+1201.067049024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/252fe161-b368-435b-8c08-f8e2b241753a-maas-api-tls") pod "maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" (UID: "252fe161-b368-435b-8c08-f8e2b241753a") : secret "maas-api-e2e-ait-b5770a0e-serving-cert" not found Nov 03 16:34:16.978560 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.978534 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/252fe161-b368-435b-8c08-f8e2b241753a-openshift-service-ca\") pod \"maas-api-e2e-ait-b5770a0e-594c557f44-fctpm\" (UID: \"252fe161-b368-435b-8c08-f8e2b241753a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:16.980993 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:16.980970 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/252fe161-b368-435b-8c08-f8e2b241753a-metrics-certs\") pod \"maas-api-e2e-ait-b5770a0e-594c557f44-fctpm\" (UID: \"252fe161-b368-435b-8c08-f8e2b241753a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:17.007113 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:17.007074 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x98cc\" (UniqueName: \"kubernetes.io/projected/252fe161-b368-435b-8c08-f8e2b241753a-kube-api-access-x98cc\") pod \"maas-api-e2e-ait-b5770a0e-594c557f44-fctpm\" (UID: \"252fe161-b368-435b-8c08-f8e2b241753a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:17.483686 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:17.483649 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/252fe161-b368-435b-8c08-f8e2b241753a-maas-api-tls\") pod \"maas-api-e2e-ait-b5770a0e-594c557f44-fctpm\" (UID: \"252fe161-b368-435b-8c08-f8e2b241753a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:17.486254 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:17.486232 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/252fe161-b368-435b-8c08-f8e2b241753a-maas-api-tls\") pod \"maas-api-e2e-ait-b5770a0e-594c557f44-fctpm\" (UID: \"252fe161-b368-435b-8c08-f8e2b241753a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:17.659604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:17.659557 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:17.715210 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:17.715168 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p" event={"ID":"2bdcb312-0f38-400e-a242-4c3d3f426179","Type":"ContainerStarted","Data":"2a814d387674f059ffb65102ee5a453103cdb33cff01e13127f30ab0c3bdd7d2"} Nov 03 16:34:17.764040 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:17.763946 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p" podStartSLOduration=2.569243292 podStartE2EDuration="4.763924896s" totalImagesPullingTime="2.194681604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:17.76258926 +0000 UTC m=+1201.351589229" watchObservedRunningTime="2026-11-03 16:34:17.763924896 +0000 UTC m=+1201.352924866" Nov 03 16:34:18.041642 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:18.037083 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm"] Nov 03 16:34:18.642301 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:18.642228 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" Nov 03 16:34:18.734999 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:18.734408 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" event={"ID":"252fe161-b368-435b-8c08-f8e2b241753a","Type":"ContainerStarted","Data":"95e9660819600523ce6d9f8d6477d342634472b4c842ab73da13c657eb58f1d0"} Nov 03 16:34:18.734999 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:18.734475 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" event={"ID":"252fe161-b368-435b-8c08-f8e2b241753a","Type":"ContainerStarted","Data":"f1c0ad7229b2b62e7dc0da58c2e1a40665f29ef878c0ab173b826aa18eac0375"} Nov 03 16:34:18.734999 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:18.734542 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p" Nov 03 16:34:18.734999 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:18.734570 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:18.784616 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:18.784558 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" podStartSLOduration=2.452945914 podStartE2EDuration="2.784545025s" totalImagesPullingTime="331.599111ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:18.783946086 +0000 UTC m=+1202.372946055" watchObservedRunningTime="2026-11-03 16:34:18.784545025 +0000 UTC m=+1202.373544994" Nov 03 16:34:20.662055 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:20.662005 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" Nov 03 16:34:21.967495 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:21.967445 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:34:22.739725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:22.738881 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" Nov 03 16:34:24.762417 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:24.762380 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:25.654907 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:25.654867 4947 patch_prober.go:29] interesting pod/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:34:25.655123 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:25.654944 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" podUID="df03b919-6262-4fd9-af88-788700865ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:34:27.920920 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:27.920842 4947 generic.go:356] "Generic (PLEG): container finished" podID="df03b919-6262-4fd9-af88-788700865ebd" containerID="f4ad9579bc3c016d75dc2032b2d60887c6cb8a76bbb54bd419bcacb95f0ed15e" exitCode=0 Nov 03 16:34:27.921326 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:27.920919 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" event={"ID":"df03b919-6262-4fd9-af88-788700865ebd","Type":"ContainerDied","Data":"f4ad9579bc3c016d75dc2032b2d60887c6cb8a76bbb54bd419bcacb95f0ed15e"} Nov 03 16:34:27.921326 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:27.921292 4947 scope.go:122] "RemoveContainer" containerID="f4ad9579bc3c016d75dc2032b2d60887c6cb8a76bbb54bd419bcacb95f0ed15e" Nov 03 16:34:27.922560 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:27.922531 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:27.922648 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:27.922598 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:27.922648 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:27.922628 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:28.345479 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.345406 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q"] Nov 03 16:34:28.348123 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.348106 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:28.364618 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.364590 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-4d5492de-serving-cert\"" Nov 03 16:34:28.389186 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.389148 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q"] Nov 03 16:34:28.433184 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.433143 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-maas-api-tls\") pod \"maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q\" (UID: \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:28.433560 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.433536 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-openshift-service-ca\") pod \"maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q\" (UID: \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:28.433799 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.433778 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-metrics-certs\") pod \"maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q\" (UID: \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:28.434055 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.434028 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbh5t\" (UniqueName: \"kubernetes.io/projected/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-kube-api-access-rbh5t\") pod \"maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q\" (UID: \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:28.535601 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.535562 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-maas-api-tls\") pod \"maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q\" (UID: \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:28.535601 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.535606 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-openshift-service-ca\") pod \"maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q\" (UID: \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:28.535867 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.535636 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-metrics-certs\") pod \"maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q\" (UID: \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:28.535867 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.535774 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbh5t\" (UniqueName: \"kubernetes.io/projected/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-kube-api-access-rbh5t\") pod \"maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q\" (UID: \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:28.535867 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:34:28.535785 4947 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-serving-cert: secret "maas-api-e2e-derive-4d5492de-serving-cert" not found Nov 03 16:34:28.535867 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:34:28.535862 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-maas-api-tls podName:9edd4279-7a2a-4f62-a819-b5d9a9c1e848 nodeName:}" failed. No retries permitted until 2026-11-03 16:34:29.03584018 +0000 UTC m=+1212.624840133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-maas-api-tls") pod "maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" (UID: "9edd4279-7a2a-4f62-a819-b5d9a9c1e848") : secret "maas-api-e2e-derive-4d5492de-serving-cert" not found Nov 03 16:34:28.536464 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.536401 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-openshift-service-ca\") pod \"maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q\" (UID: \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:28.538379 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.538346 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-metrics-certs\") pod \"maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q\" (UID: \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:28.566294 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.566267 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbh5t\" (UniqueName: \"kubernetes.io/projected/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-kube-api-access-rbh5t\") pod \"maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q\" (UID: \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:28.929351 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.929313 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" event={"ID":"df03b919-6262-4fd9-af88-788700865ebd","Type":"ContainerStarted","Data":"ca306f1148cd69a08d98e38b272b7e3c3a222beb3c5fcb2039aea5402fa24df4"} Nov 03 16:34:28.931233 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.931209 4947 generic.go:356] "Generic (PLEG): container finished" podID="6bc977de-5e83-4494-a9aa-1214a2cd29e1" containerID="c228478adf988b477f285ce01dc6f939fd3495e61884f6c0f236bf9100ce624e" exitCode=0 Nov 03 16:34:28.931359 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.931285 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" event={"ID":"6bc977de-5e83-4494-a9aa-1214a2cd29e1","Type":"ContainerDied","Data":"c228478adf988b477f285ce01dc6f939fd3495e61884f6c0f236bf9100ce624e"} Nov 03 16:34:28.931644 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.931630 4947 scope.go:122] "RemoveContainer" containerID="c228478adf988b477f285ce01dc6f939fd3495e61884f6c0f236bf9100ce624e" Nov 03 16:34:28.932828 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.932803 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:28.932884 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.932866 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:28.932918 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:28.932897 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:29.042515 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:29.042477 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-maas-api-tls\") pod \"maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q\" (UID: \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:29.046564 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:29.046536 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-maas-api-tls\") pod \"maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q\" (UID: \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:29.261726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:29.260247 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:29.623076 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:29.622255 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q"] Nov 03 16:34:29.760870 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:29.760837 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p" Nov 03 16:34:29.830723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:29.829890 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:29.830723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:29.830017 4947 patch_prober.go:29] interesting pod/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:34:29.830723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:29.830066 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" podUID="df03b919-6262-4fd9-af88-788700865ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 16:34:29.954620 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:29.954534 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" event={"ID":"6bc977de-5e83-4494-a9aa-1214a2cd29e1","Type":"ContainerStarted","Data":"fdf5c998ef82428f88d02491149d573d95c7cc37434e1ac866ca9cfaf30fbbc7"} Nov 03 16:34:29.957154 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:29.957120 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" event={"ID":"9edd4279-7a2a-4f62-a819-b5d9a9c1e848","Type":"ContainerStarted","Data":"e18edf7fd081060f578afcd9d4871e478c1d60bec56defdbcab5fb7c3e52826a"} Nov 03 16:34:30.183347 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:34:30.183304 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:34:30.654489 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:30.654453 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:30.655757 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:30.654818 4947 patch_prober.go:29] interesting pod/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 03 16:34:30.662308 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:30.662270 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" podUID="6bc977de-5e83-4494-a9aa-1214a2cd29e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 03 16:34:30.836732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:30.834714 4947 patch_prober.go:29] interesting pod/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:34:30.849394 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:30.848770 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" podUID="df03b919-6262-4fd9-af88-788700865ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 16:34:30.964156 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:30.964054 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" event={"ID":"9edd4279-7a2a-4f62-a819-b5d9a9c1e848","Type":"ContainerStarted","Data":"f6bcfcfb60e4ec59d84f691da922f52d2ba8d94a7084fd2cffb3fa13cb85538d"} Nov 03 16:34:30.964651 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:30.964323 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:31.023725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:31.020845 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" podStartSLOduration=2.711512472 podStartE2EDuration="3.020823882s" totalImagesPullingTime="309.31141ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:31.019176669 +0000 UTC m=+1214.608176638" watchObservedRunningTime="2026-11-03 16:34:31.020823882 +0000 UTC m=+1214.609823855" Nov 03 16:34:31.647206 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:31.647164 4947 patch_prober.go:29] interesting pod/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 03 16:34:31.647632 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:31.647600 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" podUID="6bc977de-5e83-4494-a9aa-1214a2cd29e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 03 16:34:31.820144 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:31.820104 4947 patch_prober.go:29] interesting pod/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:34:31.820434 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:31.820403 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" podUID="df03b919-6262-4fd9-af88-788700865ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 16:34:32.581848 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:32.581803 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:34:32.647154 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:32.647113 4947 patch_prober.go:29] interesting pod/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 03 16:34:32.647459 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:32.647430 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" podUID="6bc977de-5e83-4494-a9aa-1214a2cd29e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 03 16:34:32.820169 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:32.820128 4947 patch_prober.go:29] interesting pod/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:34:32.820483 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:32.820450 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" podUID="df03b919-6262-4fd9-af88-788700865ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 16:34:33.650723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:33.650284 4947 patch_prober.go:29] interesting pod/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 03 16:34:33.650723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:33.650346 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" podUID="6bc977de-5e83-4494-a9aa-1214a2cd29e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 03 16:34:33.823148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:33.823110 4947 patch_prober.go:29] interesting pod/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 16:34:33.823469 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:33.823439 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" podUID="df03b919-6262-4fd9-af88-788700865ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 16:34:34.644458 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:34.644418 4947 patch_prober.go:29] interesting pod/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Nov 03 16:34:34.644667 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:34.644474 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" podUID="6bc977de-5e83-4494-a9aa-1214a2cd29e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Nov 03 16:34:34.826020 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:34.825988 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:34.826465 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:34.826255 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:34.827099 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:34.827081 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:35.655980 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:35.655945 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:35.657357 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:35.657330 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:35.658251 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:35.658226 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:36.975340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:36.975312 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:42.105253 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:42.105218 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p"] Nov 03 16:34:42.433612 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:42.433530 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s"] Nov 03 16:34:42.433820 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:42.433763 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" podUID="9b6bcd10-458d-45f7-bfd7-729e65ad9cf8" containerName="maas-api" containerID="cri-o://52e0e174a7cb554f5b99c121c58e184d6009abcce506ea82091604f0c2f4bd37" gracePeriod=30 Nov 03 16:34:42.693028 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:42.692968 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" Nov 03 16:34:42.800283 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:42.792184 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-openshift-service-ca\") pod \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\" (UID: \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\") " Nov 03 16:34:42.800283 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:42.792256 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-kube-api-access-28s49\" (UniqueName: \"kubernetes.io/projected/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-kube-api-access-28s49\") pod \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\" (UID: \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\") " Nov 03 16:34:42.800283 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:42.792356 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-metrics-certs\") pod \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\" (UID: \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\") " Nov 03 16:34:42.800283 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:42.792421 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-maas-api-tls\") pod \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\" (UID: \"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8\") " Nov 03 16:34:42.800283 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:42.794677 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-openshift-service-ca" pod "9b6bcd10-458d-45f7-bfd7-729e65ad9cf8" (UID: "9b6bcd10-458d-45f7-bfd7-729e65ad9cf8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:42.809804 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:42.804742 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-kube-api-access-28s49" pod "9b6bcd10-458d-45f7-bfd7-729e65ad9cf8" (UID: "9b6bcd10-458d-45f7-bfd7-729e65ad9cf8"). InnerVolumeSpecName "kube-api-access-28s49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:42.809804 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:42.807064 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-maas-api-tls" pod "9b6bcd10-458d-45f7-bfd7-729e65ad9cf8" (UID: "9b6bcd10-458d-45f7-bfd7-729e65ad9cf8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:34:42.817782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:42.817749 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-metrics-certs" pod "9b6bcd10-458d-45f7-bfd7-729e65ad9cf8" (UID: "9b6bcd10-458d-45f7-bfd7-729e65ad9cf8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:34:42.821844 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:42.821817 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx"] Nov 03 16:34:42.822081 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:42.822052 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" podUID="6be3a9c7-cec4-46b2-87cd-09f44f31afc2" containerName="maas-api" containerID="cri-o://bcaade0418ac96f688a63a9da29810eb0c19d2af75ac1156bb46e653de6799dc" gracePeriod=30 Nov 03 16:34:42.899691 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:42.899649 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:42.899691 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:42.899690 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28s49\" (UniqueName: \"kubernetes.io/projected/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-kube-api-access-28s49\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:42.899927 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:42.899724 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:42.899927 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:42.899739 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:43.029949 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.029919 4947 generic.go:356] "Generic (PLEG): container finished" podID="6be3a9c7-cec4-46b2-87cd-09f44f31afc2" containerID="bcaade0418ac96f688a63a9da29810eb0c19d2af75ac1156bb46e653de6799dc" exitCode=0 Nov 03 16:34:43.030119 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.029984 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" event={"ID":"6be3a9c7-cec4-46b2-87cd-09f44f31afc2","Type":"ContainerDied","Data":"bcaade0418ac96f688a63a9da29810eb0c19d2af75ac1156bb46e653de6799dc"} Nov 03 16:34:43.031491 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.031467 4947 generic.go:356] "Generic (PLEG): container finished" podID="9b6bcd10-458d-45f7-bfd7-729e65ad9cf8" containerID="52e0e174a7cb554f5b99c121c58e184d6009abcce506ea82091604f0c2f4bd37" exitCode=0 Nov 03 16:34:43.031596 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.031567 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" event={"ID":"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8","Type":"ContainerDied","Data":"52e0e174a7cb554f5b99c121c58e184d6009abcce506ea82091604f0c2f4bd37"} Nov 03 16:34:43.031680 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.031609 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" event={"ID":"9b6bcd10-458d-45f7-bfd7-729e65ad9cf8","Type":"ContainerDied","Data":"5980315c674f9be58e6089e0afd90dd58669f42d2c8a82ebfdc6447aeeb17a7b"} Nov 03 16:34:43.031680 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.031642 4947 scope.go:122] "RemoveContainer" containerID="52e0e174a7cb554f5b99c121c58e184d6009abcce506ea82091604f0c2f4bd37" Nov 03 16:34:43.031844 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.031824 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s" Nov 03 16:34:43.050961 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.047277 4947 scope.go:122] "RemoveContainer" containerID="52e0e174a7cb554f5b99c121c58e184d6009abcce506ea82091604f0c2f4bd37" Nov 03 16:34:43.050961 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:34:43.050932 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52e0e174a7cb554f5b99c121c58e184d6009abcce506ea82091604f0c2f4bd37\": container with ID starting with 52e0e174a7cb554f5b99c121c58e184d6009abcce506ea82091604f0c2f4bd37 not found: ID does not exist" containerID="52e0e174a7cb554f5b99c121c58e184d6009abcce506ea82091604f0c2f4bd37" Nov 03 16:34:43.051106 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.050968 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52e0e174a7cb554f5b99c121c58e184d6009abcce506ea82091604f0c2f4bd37"} err="failed to get container status \"52e0e174a7cb554f5b99c121c58e184d6009abcce506ea82091604f0c2f4bd37\": rpc error: code = NotFound desc = could not find container \"52e0e174a7cb554f5b99c121c58e184d6009abcce506ea82091604f0c2f4bd37\": container with ID starting with 52e0e174a7cb554f5b99c121c58e184d6009abcce506ea82091604f0c2f4bd37 not found: ID does not exist" Nov 03 16:34:43.089341 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.089310 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s"] Nov 03 16:34:43.096425 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.096395 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-cacca478-bcc47584-thd5s"] Nov 03 16:34:43.110817 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.110795 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" Nov 03 16:34:43.137654 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.137628 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5"] Nov 03 16:34:43.137887 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.137847 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" podUID="a8cc265f-63b8-4014-a19f-708f8c38d6a7" containerName="maas-api" containerID="cri-o://83dbd7290bb37b6ab476149a3d85fafdcc4c61042d4b24ee35882331913501fb" gracePeriod=30 Nov 03 16:34:43.204177 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.204142 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-kube-api-access-42jc4\" (UniqueName: \"kubernetes.io/projected/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-kube-api-access-42jc4\") pod \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\" (UID: \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\") " Nov 03 16:34:43.204365 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.204263 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-openshift-service-ca\") pod \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\" (UID: \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\") " Nov 03 16:34:43.204432 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.204403 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-metrics-certs\") pod \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\" (UID: \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\") " Nov 03 16:34:43.204508 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.204491 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-maas-api-tls\") pod \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\" (UID: \"6be3a9c7-cec4-46b2-87cd-09f44f31afc2\") " Nov 03 16:34:43.205009 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.204963 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-openshift-service-ca" pod "6be3a9c7-cec4-46b2-87cd-09f44f31afc2" (UID: "6be3a9c7-cec4-46b2-87cd-09f44f31afc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:43.208195 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.208156 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-maas-api-tls" pod "6be3a9c7-cec4-46b2-87cd-09f44f31afc2" (UID: "6be3a9c7-cec4-46b2-87cd-09f44f31afc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:34:43.208908 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.208885 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-kube-api-access-42jc4" pod "6be3a9c7-cec4-46b2-87cd-09f44f31afc2" (UID: "6be3a9c7-cec4-46b2-87cd-09f44f31afc2"). InnerVolumeSpecName "kube-api-access-42jc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:43.209483 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.209457 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-metrics-certs" pod "6be3a9c7-cec4-46b2-87cd-09f44f31afc2" (UID: "6be3a9c7-cec4-46b2-87cd-09f44f31afc2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:34:43.306066 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.305983 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:43.306066 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.306014 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42jc4\" (UniqueName: \"kubernetes.io/projected/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-kube-api-access-42jc4\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:43.306066 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.306025 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:43.306066 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.306036 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6be3a9c7-cec4-46b2-87cd-09f44f31afc2-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:43.386584 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.386562 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" Nov 03 16:34:43.406866 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.406835 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8cc265f-63b8-4014-a19f-708f8c38d6a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cc265f-63b8-4014-a19f-708f8c38d6a7-openshift-service-ca\") pod \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\" (UID: \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\") " Nov 03 16:34:43.407019 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.406915 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8cc265f-63b8-4014-a19f-708f8c38d6a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cc265f-63b8-4014-a19f-708f8c38d6a7-maas-api-tls\") pod \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\" (UID: \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\") " Nov 03 16:34:43.407019 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.406975 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8cc265f-63b8-4014-a19f-708f8c38d6a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cc265f-63b8-4014-a19f-708f8c38d6a7-metrics-certs\") pod \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\" (UID: \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\") " Nov 03 16:34:43.407149 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.407040 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8cc265f-63b8-4014-a19f-708f8c38d6a7-kube-api-access-2qwlz\" (UniqueName: \"kubernetes.io/projected/a8cc265f-63b8-4014-a19f-708f8c38d6a7-kube-api-access-2qwlz\") pod \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\" (UID: \"a8cc265f-63b8-4014-a19f-708f8c38d6a7\") " Nov 03 16:34:43.407253 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.407226 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8cc265f-63b8-4014-a19f-708f8c38d6a7-openshift-service-ca" pod "a8cc265f-63b8-4014-a19f-708f8c38d6a7" (UID: "a8cc265f-63b8-4014-a19f-708f8c38d6a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:43.407359 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.407343 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8cc265f-63b8-4014-a19f-708f8c38d6a7-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:43.409299 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.409274 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8cc265f-63b8-4014-a19f-708f8c38d6a7-metrics-certs" pod "a8cc265f-63b8-4014-a19f-708f8c38d6a7" (UID: "a8cc265f-63b8-4014-a19f-708f8c38d6a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:34:43.409447 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.409428 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8cc265f-63b8-4014-a19f-708f8c38d6a7-maas-api-tls" pod "a8cc265f-63b8-4014-a19f-708f8c38d6a7" (UID: "a8cc265f-63b8-4014-a19f-708f8c38d6a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:34:43.409520 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.409481 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8cc265f-63b8-4014-a19f-708f8c38d6a7-kube-api-access-2qwlz" pod "a8cc265f-63b8-4014-a19f-708f8c38d6a7" (UID: "a8cc265f-63b8-4014-a19f-708f8c38d6a7"). InnerVolumeSpecName "kube-api-access-2qwlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:43.508584 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.508545 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8cc265f-63b8-4014-a19f-708f8c38d6a7-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:43.508584 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.508577 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qwlz\" (UniqueName: \"kubernetes.io/projected/a8cc265f-63b8-4014-a19f-708f8c38d6a7-kube-api-access-2qwlz\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:43.508584 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.508590 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8cc265f-63b8-4014-a19f-708f8c38d6a7-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:43.525821 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.524132 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm"] Nov 03 16:34:43.525821 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.524429 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" podUID="252fe161-b368-435b-8c08-f8e2b241753a" containerName="maas-api" containerID="cri-o://95e9660819600523ce6d9f8d6477d342634472b4c842ab73da13c657eb58f1d0" gracePeriod=30 Nov 03 16:34:43.783733 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.783688 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:43.811862 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.811830 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/252fe161-b368-435b-8c08-f8e2b241753a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/252fe161-b368-435b-8c08-f8e2b241753a-metrics-certs\") pod \"252fe161-b368-435b-8c08-f8e2b241753a\" (UID: \"252fe161-b368-435b-8c08-f8e2b241753a\") " Nov 03 16:34:43.812016 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.811887 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/252fe161-b368-435b-8c08-f8e2b241753a-kube-api-access-x98cc\" (UniqueName: \"kubernetes.io/projected/252fe161-b368-435b-8c08-f8e2b241753a-kube-api-access-x98cc\") pod \"252fe161-b368-435b-8c08-f8e2b241753a\" (UID: \"252fe161-b368-435b-8c08-f8e2b241753a\") " Nov 03 16:34:43.812016 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.811990 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/252fe161-b368-435b-8c08-f8e2b241753a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/252fe161-b368-435b-8c08-f8e2b241753a-openshift-service-ca\") pod \"252fe161-b368-435b-8c08-f8e2b241753a\" (UID: \"252fe161-b368-435b-8c08-f8e2b241753a\") " Nov 03 16:34:43.812170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.812106 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/252fe161-b368-435b-8c08-f8e2b241753a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/252fe161-b368-435b-8c08-f8e2b241753a-maas-api-tls\") pod \"252fe161-b368-435b-8c08-f8e2b241753a\" (UID: \"252fe161-b368-435b-8c08-f8e2b241753a\") " Nov 03 16:34:43.812433 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.812391 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252fe161-b368-435b-8c08-f8e2b241753a-openshift-service-ca" pod "252fe161-b368-435b-8c08-f8e2b241753a" (UID: "252fe161-b368-435b-8c08-f8e2b241753a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:43.814345 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.814314 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/252fe161-b368-435b-8c08-f8e2b241753a-metrics-certs" pod "252fe161-b368-435b-8c08-f8e2b241753a" (UID: "252fe161-b368-435b-8c08-f8e2b241753a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:34:43.814490 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.814472 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252fe161-b368-435b-8c08-f8e2b241753a-kube-api-access-x98cc" pod "252fe161-b368-435b-8c08-f8e2b241753a" (UID: "252fe161-b368-435b-8c08-f8e2b241753a"). InnerVolumeSpecName "kube-api-access-x98cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:43.814808 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.814774 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/252fe161-b368-435b-8c08-f8e2b241753a-maas-api-tls" pod "252fe161-b368-435b-8c08-f8e2b241753a" (UID: "252fe161-b368-435b-8c08-f8e2b241753a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:34:43.913340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.913259 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/252fe161-b368-435b-8c08-f8e2b241753a-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:43.913340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.913287 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/252fe161-b368-435b-8c08-f8e2b241753a-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:43.913340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.913296 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/252fe161-b368-435b-8c08-f8e2b241753a-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:43.913340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:43.913305 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x98cc\" (UniqueName: \"kubernetes.io/projected/252fe161-b368-435b-8c08-f8e2b241753a-kube-api-access-x98cc\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:44.041207 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.041164 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w"] Nov 03 16:34:44.041419 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.041366 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" podUID="6bc977de-5e83-4494-a9aa-1214a2cd29e1" containerName="istio-proxy" containerID="cri-o://fdf5c998ef82428f88d02491149d573d95c7cc37434e1ac866ca9cfaf30fbbc7" gracePeriod=30 Nov 03 16:34:44.044271 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.043994 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" event={"ID":"6be3a9c7-cec4-46b2-87cd-09f44f31afc2","Type":"ContainerDied","Data":"7657d77eaa62ba2d10fd2be68eec150fdeb121816a8e573c7c23055048bf3dc3"} Nov 03 16:34:44.044271 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.044068 4947 scope.go:122] "RemoveContainer" containerID="bcaade0418ac96f688a63a9da29810eb0c19d2af75ac1156bb46e653de6799dc" Nov 03 16:34:44.045456 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.045432 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx" Nov 03 16:34:44.045946 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.045826 4947 generic.go:356] "Generic (PLEG): container finished" podID="a8cc265f-63b8-4014-a19f-708f8c38d6a7" containerID="83dbd7290bb37b6ab476149a3d85fafdcc4c61042d4b24ee35882331913501fb" exitCode=0 Nov 03 16:34:44.046016 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.045976 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" event={"ID":"a8cc265f-63b8-4014-a19f-708f8c38d6a7","Type":"ContainerDied","Data":"83dbd7290bb37b6ab476149a3d85fafdcc4c61042d4b24ee35882331913501fb"} Nov 03 16:34:44.046070 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.046020 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" event={"ID":"a8cc265f-63b8-4014-a19f-708f8c38d6a7","Type":"ContainerDied","Data":"a0c36c54ffde7d69ae132e249fe2dec3ca3482c8351a56f7aee0d4a5563c5f00"} Nov 03 16:34:44.046132 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.046099 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5" Nov 03 16:34:44.048863 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.048760 4947 generic.go:356] "Generic (PLEG): container finished" podID="252fe161-b368-435b-8c08-f8e2b241753a" containerID="95e9660819600523ce6d9f8d6477d342634472b4c842ab73da13c657eb58f1d0" exitCode=0 Nov 03 16:34:44.048863 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.048802 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" Nov 03 16:34:44.048863 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.048846 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" event={"ID":"252fe161-b368-435b-8c08-f8e2b241753a","Type":"ContainerDied","Data":"95e9660819600523ce6d9f8d6477d342634472b4c842ab73da13c657eb58f1d0"} Nov 03 16:34:44.049077 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.048903 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm" event={"ID":"252fe161-b368-435b-8c08-f8e2b241753a","Type":"ContainerDied","Data":"f1c0ad7229b2b62e7dc0da58c2e1a40665f29ef878c0ab173b826aa18eac0375"} Nov 03 16:34:44.065962 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.065937 4947 scope.go:122] "RemoveContainer" containerID="83dbd7290bb37b6ab476149a3d85fafdcc4c61042d4b24ee35882331913501fb" Nov 03 16:34:44.074942 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.074920 4947 scope.go:122] "RemoveContainer" containerID="83dbd7290bb37b6ab476149a3d85fafdcc4c61042d4b24ee35882331913501fb" Nov 03 16:34:44.075240 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:34:44.075222 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83dbd7290bb37b6ab476149a3d85fafdcc4c61042d4b24ee35882331913501fb\": container with ID starting with 83dbd7290bb37b6ab476149a3d85fafdcc4c61042d4b24ee35882331913501fb not found: ID does not exist" containerID="83dbd7290bb37b6ab476149a3d85fafdcc4c61042d4b24ee35882331913501fb" Nov 03 16:34:44.075281 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.075251 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83dbd7290bb37b6ab476149a3d85fafdcc4c61042d4b24ee35882331913501fb"} err="failed to get container status \"83dbd7290bb37b6ab476149a3d85fafdcc4c61042d4b24ee35882331913501fb\": rpc error: code = NotFound desc = could not find container \"83dbd7290bb37b6ab476149a3d85fafdcc4c61042d4b24ee35882331913501fb\": container with ID starting with 83dbd7290bb37b6ab476149a3d85fafdcc4c61042d4b24ee35882331913501fb not found: ID does not exist" Nov 03 16:34:44.075281 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.075270 4947 scope.go:122] "RemoveContainer" containerID="95e9660819600523ce6d9f8d6477d342634472b4c842ab73da13c657eb58f1d0" Nov 03 16:34:44.083788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.083765 4947 scope.go:122] "RemoveContainer" containerID="95e9660819600523ce6d9f8d6477d342634472b4c842ab73da13c657eb58f1d0" Nov 03 16:34:44.084198 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:34:44.084174 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e9660819600523ce6d9f8d6477d342634472b4c842ab73da13c657eb58f1d0\": container with ID starting with 95e9660819600523ce6d9f8d6477d342634472b4c842ab73da13c657eb58f1d0 not found: ID does not exist" containerID="95e9660819600523ce6d9f8d6477d342634472b4c842ab73da13c657eb58f1d0" Nov 03 16:34:44.084293 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.084209 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e9660819600523ce6d9f8d6477d342634472b4c842ab73da13c657eb58f1d0"} err="failed to get container status \"95e9660819600523ce6d9f8d6477d342634472b4c842ab73da13c657eb58f1d0\": rpc error: code = NotFound desc = could not find container \"95e9660819600523ce6d9f8d6477d342634472b4c842ab73da13c657eb58f1d0\": container with ID starting with 95e9660819600523ce6d9f8d6477d342634472b4c842ab73da13c657eb58f1d0 not found: ID does not exist" Nov 03 16:34:44.129090 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.129057 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx"] Nov 03 16:34:44.136361 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.136310 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0b922f4-784778d58-dc9zx"] Nov 03 16:34:44.186674 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.186489 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm"] Nov 03 16:34:44.193501 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.193473 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b5770a0e-594c557f44-fctpm"] Nov 03 16:34:44.229188 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.229151 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5"] Nov 03 16:34:44.238412 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.238385 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c9fa1268-6d55786687-4hfd5"] Nov 03 16:34:44.515680 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.515643 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755"] Nov 03 16:34:44.516099 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.516082 4947 cpu_manager.go:414] "containerMap: removing container" podUID="252fe161-b368-435b-8c08-f8e2b241753a" containerName="maas-api" Nov 03 16:34:44.516099 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.516098 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="252fe161-b368-435b-8c08-f8e2b241753a" containerName="maas-api" Nov 03 16:34:44.516258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.516108 4947 cpu_manager.go:414] "containerMap: removing container" podUID="a8cc265f-63b8-4014-a19f-708f8c38d6a7" containerName="maas-api" Nov 03 16:34:44.516258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.516113 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8cc265f-63b8-4014-a19f-708f8c38d6a7" containerName="maas-api" Nov 03 16:34:44.516258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.516138 4947 cpu_manager.go:414] "containerMap: removing container" podUID="6be3a9c7-cec4-46b2-87cd-09f44f31afc2" containerName="maas-api" Nov 03 16:34:44.516258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.516143 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be3a9c7-cec4-46b2-87cd-09f44f31afc2" containerName="maas-api" Nov 03 16:34:44.516258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.516156 4947 cpu_manager.go:414] "containerMap: removing container" podUID="9b6bcd10-458d-45f7-bfd7-729e65ad9cf8" containerName="maas-api" Nov 03 16:34:44.516258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.516161 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b6bcd10-458d-45f7-bfd7-729e65ad9cf8" containerName="maas-api" Nov 03 16:34:44.516258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.516218 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="252fe161-b368-435b-8c08-f8e2b241753a" containerName="maas-api" Nov 03 16:34:44.516258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.516224 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8cc265f-63b8-4014-a19f-708f8c38d6a7" containerName="maas-api" Nov 03 16:34:44.516258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.516235 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be3a9c7-cec4-46b2-87cd-09f44f31afc2" containerName="maas-api" Nov 03 16:34:44.516258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.516243 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b6bcd10-458d-45f7-bfd7-729e65ad9cf8" containerName="maas-api" Nov 03 16:34:44.518281 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.518260 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.539500 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.539471 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3a2fa2-openshift-default-dockercfg-q9n76\"" Nov 03 16:34:44.548958 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.548923 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755"] Nov 03 16:34:44.565613 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.565583 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="252fe161-b368-435b-8c08-f8e2b241753a" path="/var/lib/kubelet/pods/252fe161-b368-435b-8c08-f8e2b241753a/volumes" Nov 03 16:34:44.566014 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.566000 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be3a9c7-cec4-46b2-87cd-09f44f31afc2" path="/var/lib/kubelet/pods/6be3a9c7-cec4-46b2-87cd-09f44f31afc2/volumes" Nov 03 16:34:44.566344 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.566333 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b6bcd10-458d-45f7-bfd7-729e65ad9cf8" path="/var/lib/kubelet/pods/9b6bcd10-458d-45f7-bfd7-729e65ad9cf8/volumes" Nov 03 16:34:44.566836 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.566817 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8cc265f-63b8-4014-a19f-708f8c38d6a7" path="/var/lib/kubelet/pods/a8cc265f-63b8-4014-a19f-708f8c38d6a7/volumes" Nov 03 16:34:44.620816 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.620774 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.620816 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.620813 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.621069 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.620834 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.621069 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.620854 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbbtm\" (UniqueName: \"kubernetes.io/projected/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-kube-api-access-wbbtm\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.621069 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.620881 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.621290 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.621269 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.621646 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.621301 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.621646 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.621396 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.621646 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.621422 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.723165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.723126 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.723370 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.723181 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.723370 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.723214 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.723370 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.723247 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbbtm\" (UniqueName: \"kubernetes.io/projected/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-kube-api-access-wbbtm\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.723666 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.723644 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.723790 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.723674 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.723790 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.723751 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.723790 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.723781 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.723947 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.723839 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.723947 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.723869 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.724059 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.723644 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.724200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.724181 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.724452 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.724428 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.724931 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.724907 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.726992 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.726954 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.727600 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.727576 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.751804 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.751773 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbbtm\" (UniqueName: \"kubernetes.io/projected/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-kube-api-access-wbbtm\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.753318 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.753282 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:44.832218 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:44.832129 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:45.045571 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.045539 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2"] Nov 03 16:34:45.045846 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.045819 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" podUID="df03b919-6262-4fd9-af88-788700865ebd" containerName="istio-proxy" containerID="cri-o://ca306f1148cd69a08d98e38b272b7e3c3a222beb3c5fcb2039aea5402fa24df4" gracePeriod=30 Nov 03 16:34:45.167831 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.166324 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755"] Nov 03 16:34:45.187468 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.187420 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:45.189568 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.187517 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:45.189568 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.187557 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:45.305276 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.305242 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7"] Nov 03 16:34:45.308197 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.308169 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.334372 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.334341 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79da839f-f698-42eb-9275-9999cb9a1306-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.334616 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.334586 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79da839f-f698-42eb-9275-9999cb9a1306-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.334756 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.334655 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.334817 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.334790 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.334868 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.334827 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.334945 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.334927 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79da839f-f698-42eb-9275-9999cb9a1306-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.335016 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.334988 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.335191 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.335166 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7"] Nov 03 16:34:45.335309 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.335286 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmm8r\" (UniqueName: \"kubernetes.io/projected/79da839f-f698-42eb-9275-9999cb9a1306-kube-api-access-cmm8r\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.335410 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.335394 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.380730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.380640 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-7519b7-openshift-default-dockercfg-dv7k2\"" Nov 03 16:34:45.436917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.436813 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.436917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.436879 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.437151 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.436941 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79da839f-f698-42eb-9275-9999cb9a1306-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.437151 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.436984 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.437151 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.437016 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmm8r\" (UniqueName: \"kubernetes.io/projected/79da839f-f698-42eb-9275-9999cb9a1306-kube-api-access-cmm8r\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.437151 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.437063 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.437151 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.437132 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79da839f-f698-42eb-9275-9999cb9a1306-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.437391 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.437160 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79da839f-f698-42eb-9275-9999cb9a1306-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.437391 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.437219 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.437391 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.437292 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.437391 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.437342 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.437997 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.437977 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79da839f-f698-42eb-9275-9999cb9a1306-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.438293 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.438270 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.438349 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.438290 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.440453 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.440433 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.440548 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.440444 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79da839f-f698-42eb-9275-9999cb9a1306-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.462550 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.462519 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79da839f-f698-42eb-9275-9999cb9a1306-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.463171 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.463144 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmm8r\" (UniqueName: \"kubernetes.io/projected/79da839f-f698-42eb-9275-9999cb9a1306-kube-api-access-cmm8r\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.647394 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.647360 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:45.971809 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.971776 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7"] Nov 03 16:34:45.971962 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:34:45.971852 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79da839f_f698_42eb_9275_9999cb9a1306.slice/crio-1632baf4a5e9fabae6f8f561f7f57b7dd7c11a2c5fe39dd9eafbd02e744fb643 WatchSource:0}: Error finding container 1632baf4a5e9fabae6f8f561f7f57b7dd7c11a2c5fe39dd9eafbd02e744fb643: Status 404 returned error can't find the container with id 1632baf4a5e9fabae6f8f561f7f57b7dd7c11a2c5fe39dd9eafbd02e744fb643 Nov 03 16:34:45.974357 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.974318 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:45.974426 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.974405 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:45.974469 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:45.974445 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:46.065002 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.064960 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" event={"ID":"79da839f-f698-42eb-9275-9999cb9a1306","Type":"ContainerStarted","Data":"dfd75461b6ce7d634582b29fe24a6e8fa4f494a0780227ecd729038ccaa03a2f"} Nov 03 16:34:46.065186 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.065032 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" event={"ID":"79da839f-f698-42eb-9275-9999cb9a1306","Type":"ContainerStarted","Data":"1632baf4a5e9fabae6f8f561f7f57b7dd7c11a2c5fe39dd9eafbd02e744fb643"} Nov 03 16:34:46.067020 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.066993 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" event={"ID":"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20","Type":"ContainerStarted","Data":"1bd0a0d962da6a0aa8a1b0b5ddd7aea25febefee980fe34c58319351bc291dcb"} Nov 03 16:34:46.067140 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.067038 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" event={"ID":"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20","Type":"ContainerStarted","Data":"85b3b993caf02578af3d8fb9994d5ddf34e6d70573feff0d72349db781f9b2de"} Nov 03 16:34:46.115540 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.115488 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" podStartSLOduration=1.1154728839999999 podStartE2EDuration="1.115472884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:46.114223036 +0000 UTC m=+1229.703223004" watchObservedRunningTime="2026-11-03 16:34:46.115472884 +0000 UTC m=+1229.704472852" Nov 03 16:34:46.166525 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.166464 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" podStartSLOduration=2.166448494 podStartE2EDuration="2.166448494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:46.164206126 +0000 UTC m=+1229.753206094" watchObservedRunningTime="2026-11-03 16:34:46.166448494 +0000 UTC m=+1229.755448463" Nov 03 16:34:46.270955 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.270924 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:34:46.323630 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.323592 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p"] Nov 03 16:34:46.327954 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.327929 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.348230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.347752 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc5gv\" (UniqueName: \"kubernetes.io/projected/adc797dd-c98b-4af6-8939-770401241bf0-kube-api-access-kc5gv\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.348230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.347835 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adc797dd-c98b-4af6-8939-770401241bf0-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.348230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.348039 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.348230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.348109 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.348230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.348142 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adc797dd-c98b-4af6-8939-770401241bf0-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.348819 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.348251 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.348819 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.348289 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.348819 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.348331 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.348819 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.348421 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adc797dd-c98b-4af6-8939-770401241bf0-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.351166 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.351142 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-460104-openshift-default-dockercfg-4t76j\"" Nov 03 16:34:46.354384 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.354359 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p"] Nov 03 16:34:46.449610 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.449576 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adc797dd-c98b-4af6-8939-770401241bf0-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.449841 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.449626 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc5gv\" (UniqueName: \"kubernetes.io/projected/adc797dd-c98b-4af6-8939-770401241bf0-kube-api-access-kc5gv\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.449841 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.449741 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adc797dd-c98b-4af6-8939-770401241bf0-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.449841 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.449786 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.449841 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.449834 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.450075 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.449862 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adc797dd-c98b-4af6-8939-770401241bf0-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.450075 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.449917 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.450075 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.449957 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.450075 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.450005 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.450274 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.450257 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.450440 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.450417 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.450526 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.450508 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.450634 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.450607 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adc797dd-c98b-4af6-8939-770401241bf0-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.450773 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.450617 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.452850 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.452821 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.452977 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.452958 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adc797dd-c98b-4af6-8939-770401241bf0-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.480842 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.480813 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adc797dd-c98b-4af6-8939-770401241bf0-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.480975 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.480927 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc5gv\" (UniqueName: \"kubernetes.io/projected/adc797dd-c98b-4af6-8939-770401241bf0-kube-api-access-kc5gv\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.644334 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.644254 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:46.647604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.647582 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:46.832648 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.832614 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:46.837588 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.837561 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:46.962989 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.962963 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p"] Nov 03 16:34:46.963575 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:34:46.963552 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc797dd_c98b_4af6_8939_770401241bf0.slice/crio-046c349016d0a4e10a2b19ecfb34838ba895af90b392fba6644c63841855147e WatchSource:0}: Error finding container 046c349016d0a4e10a2b19ecfb34838ba895af90b392fba6644c63841855147e: Status 404 returned error can't find the container with id 046c349016d0a4e10a2b19ecfb34838ba895af90b392fba6644c63841855147e Nov 03 16:34:46.965676 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.965646 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:46.965763 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.965728 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:46.965763 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:46.965757 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:34:47.074239 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:47.074197 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" event={"ID":"adc797dd-c98b-4af6-8939-770401241bf0","Type":"ContainerStarted","Data":"bb75213e9c1538e13c87667b0d6e2801ce0f1d4c37630a6013cf489cd90caf0d"} Nov 03 16:34:47.074439 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:47.074260 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" event={"ID":"adc797dd-c98b-4af6-8939-770401241bf0","Type":"ContainerStarted","Data":"046c349016d0a4e10a2b19ecfb34838ba895af90b392fba6644c63841855147e"} Nov 03 16:34:47.074664 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:47.074645 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:47.075570 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:47.075549 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:34:47.105639 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:47.105600 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p" podUID="2bdcb312-0f38-400e-a242-4c3d3f426179" containerName="payload-processing" containerID="cri-o://2a814d387674f059ffb65102ee5a453103cdb33cff01e13127f30ab0c3bdd7d2" gracePeriod=25 Nov 03 16:34:47.130524 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:47.130471 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" podStartSLOduration=1.130455524 podStartE2EDuration="1.130455524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:47.128629252 +0000 UTC m=+1230.717629220" watchObservedRunningTime="2026-11-03 16:34:47.130455524 +0000 UTC m=+1230.719455492" Nov 03 16:34:47.409519 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:47.409494 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p" Nov 03 16:34:47.461745 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:47.461709 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bdcb312-0f38-400e-a242-4c3d3f426179-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bdcb312-0f38-400e-a242-4c3d3f426179-plugins-config-volume\") pod \"2bdcb312-0f38-400e-a242-4c3d3f426179\" (UID: \"2bdcb312-0f38-400e-a242-4c3d3f426179\") " Nov 03 16:34:47.461903 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:47.461770 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bdcb312-0f38-400e-a242-4c3d3f426179-kube-api-access-6n9mw\" (UniqueName: \"kubernetes.io/projected/2bdcb312-0f38-400e-a242-4c3d3f426179-kube-api-access-6n9mw\") pod \"2bdcb312-0f38-400e-a242-4c3d3f426179\" (UID: \"2bdcb312-0f38-400e-a242-4c3d3f426179\") " Nov 03 16:34:47.462262 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:47.462231 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bdcb312-0f38-400e-a242-4c3d3f426179-plugins-config-volume" pod "2bdcb312-0f38-400e-a242-4c3d3f426179" (UID: "2bdcb312-0f38-400e-a242-4c3d3f426179"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:47.464427 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:47.464394 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bdcb312-0f38-400e-a242-4c3d3f426179-kube-api-access-6n9mw" pod "2bdcb312-0f38-400e-a242-4c3d3f426179" (UID: "2bdcb312-0f38-400e-a242-4c3d3f426179"). InnerVolumeSpecName "kube-api-access-6n9mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:47.563379 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:47.563340 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bdcb312-0f38-400e-a242-4c3d3f426179-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:47.563379 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:47.563371 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n9mw\" (UniqueName: \"kubernetes.io/projected/2bdcb312-0f38-400e-a242-4c3d3f426179-kube-api-access-6n9mw\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:47.644676 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:47.644634 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:47.653526 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:47.653491 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:48.080896 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:48.080854 4947 generic.go:356] "Generic (PLEG): container finished" podID="2bdcb312-0f38-400e-a242-4c3d3f426179" containerID="2a814d387674f059ffb65102ee5a453103cdb33cff01e13127f30ab0c3bdd7d2" exitCode=0 Nov 03 16:34:48.081109 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:48.080925 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p" Nov 03 16:34:48.081109 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:48.080938 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p" event={"ID":"2bdcb312-0f38-400e-a242-4c3d3f426179","Type":"ContainerDied","Data":"2a814d387674f059ffb65102ee5a453103cdb33cff01e13127f30ab0c3bdd7d2"} Nov 03 16:34:48.081109 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:48.081022 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p" event={"ID":"2bdcb312-0f38-400e-a242-4c3d3f426179","Type":"ContainerDied","Data":"915df1932ba160fa0f031d5170a0b00d33fcde84ed9fe18df5184aed97330554"} Nov 03 16:34:48.081109 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:48.081045 4947 scope.go:122] "RemoveContainer" containerID="2a814d387674f059ffb65102ee5a453103cdb33cff01e13127f30ab0c3bdd7d2" Nov 03 16:34:48.081663 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:48.081633 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:48.084825 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:48.084806 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:34:48.093223 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:48.093203 4947 scope.go:122] "RemoveContainer" containerID="2a814d387674f059ffb65102ee5a453103cdb33cff01e13127f30ab0c3bdd7d2" Nov 03 16:34:48.093562 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:34:48.093542 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a814d387674f059ffb65102ee5a453103cdb33cff01e13127f30ab0c3bdd7d2\": container with ID starting with 2a814d387674f059ffb65102ee5a453103cdb33cff01e13127f30ab0c3bdd7d2 not found: ID does not exist" containerID="2a814d387674f059ffb65102ee5a453103cdb33cff01e13127f30ab0c3bdd7d2" Nov 03 16:34:48.093619 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:48.093575 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a814d387674f059ffb65102ee5a453103cdb33cff01e13127f30ab0c3bdd7d2"} err="failed to get container status \"2a814d387674f059ffb65102ee5a453103cdb33cff01e13127f30ab0c3bdd7d2\": rpc error: code = NotFound desc = could not find container \"2a814d387674f059ffb65102ee5a453103cdb33cff01e13127f30ab0c3bdd7d2\": container with ID starting with 2a814d387674f059ffb65102ee5a453103cdb33cff01e13127f30ab0c3bdd7d2 not found: ID does not exist" Nov 03 16:34:48.135734 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:48.135680 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p"] Nov 03 16:34:48.139390 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:48.139364 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c9fa1268-5b6c7ff967-mwv6p"] Nov 03 16:34:48.564454 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:48.564419 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bdcb312-0f38-400e-a242-4c3d3f426179" path="/var/lib/kubelet/pods/2bdcb312-0f38-400e-a242-4c3d3f426179/volumes" Nov 03 16:34:48.650925 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:48.650898 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:49.089972 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.089944 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:49.091220 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.091193 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:34:49.314860 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.314839 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:49.381980 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.380138 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-workload-certs\") pod \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " Nov 03 16:34:49.381980 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.380198 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-podinfo\") pod \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " Nov 03 16:34:49.381980 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.380229 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-data\") pod \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " Nov 03 16:34:49.381980 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.380287 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc977de-5e83-4494-a9aa-1214a2cd29e1-kube-api-access-brj2w\" (UniqueName: \"kubernetes.io/projected/6bc977de-5e83-4494-a9aa-1214a2cd29e1-kube-api-access-brj2w\") pod \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " Nov 03 16:34:49.381980 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.380323 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-credential-socket\") pod \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " Nov 03 16:34:49.381980 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.380391 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-envoy\") pod \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " Nov 03 16:34:49.381980 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.380464 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istiod-ca-cert\") pod \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " Nov 03 16:34:49.381980 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.380502 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-workload-socket\") pod \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " Nov 03 16:34:49.381980 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.380539 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-token\" (UniqueName: \"kubernetes.io/projected/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-token\") pod \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\" (UID: \"6bc977de-5e83-4494-a9aa-1214a2cd29e1\") " Nov 03 16:34:49.382615 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.382159 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-credential-socket" pod "6bc977de-5e83-4494-a9aa-1214a2cd29e1" (UID: "6bc977de-5e83-4494-a9aa-1214a2cd29e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:49.383165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.383130 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-data" pod "6bc977de-5e83-4494-a9aa-1214a2cd29e1" (UID: "6bc977de-5e83-4494-a9aa-1214a2cd29e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:49.383280 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.383179 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istiod-ca-cert" pod "6bc977de-5e83-4494-a9aa-1214a2cd29e1" (UID: "6bc977de-5e83-4494-a9aa-1214a2cd29e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:49.383363 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.383334 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-workload-certs" pod "6bc977de-5e83-4494-a9aa-1214a2cd29e1" (UID: "6bc977de-5e83-4494-a9aa-1214a2cd29e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:49.383489 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.383465 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-workload-socket" pod "6bc977de-5e83-4494-a9aa-1214a2cd29e1" (UID: "6bc977de-5e83-4494-a9aa-1214a2cd29e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:49.384228 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.384203 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-token" pod "6bc977de-5e83-4494-a9aa-1214a2cd29e1" (UID: "6bc977de-5e83-4494-a9aa-1214a2cd29e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:49.384872 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.384850 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-envoy" pod "6bc977de-5e83-4494-a9aa-1214a2cd29e1" (UID: "6bc977de-5e83-4494-a9aa-1214a2cd29e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:49.384872 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.384866 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc977de-5e83-4494-a9aa-1214a2cd29e1-kube-api-access-brj2w" pod "6bc977de-5e83-4494-a9aa-1214a2cd29e1" (UID: "6bc977de-5e83-4494-a9aa-1214a2cd29e1"). InnerVolumeSpecName "kube-api-access-brj2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:49.385017 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.384909 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-podinfo" pod "6bc977de-5e83-4494-a9aa-1214a2cd29e1" (UID: "6bc977de-5e83-4494-a9aa-1214a2cd29e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:34:49.482332 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.482301 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:49.482332 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.482327 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:49.482332 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.482337 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:49.482582 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.482345 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:49.482582 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.482353 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:49.482582 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.482361 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brj2w\" (UniqueName: \"kubernetes.io/projected/6bc977de-5e83-4494-a9aa-1214a2cd29e1-kube-api-access-brj2w\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:49.482582 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.482371 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:49.482582 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.482380 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:49.482582 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.482387 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6bc977de-5e83-4494-a9aa-1214a2cd29e1-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:49.829388 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.829345 4947 patch_prober.go:29] interesting pod/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:34:49.829867 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:49.829434 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" podUID="df03b919-6262-4fd9-af88-788700865ebd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:34:50.096555 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.096462 4947 generic.go:356] "Generic (PLEG): container finished" podID="6bc977de-5e83-4494-a9aa-1214a2cd29e1" containerID="fdf5c998ef82428f88d02491149d573d95c7cc37434e1ac866ca9cfaf30fbbc7" exitCode=0 Nov 03 16:34:50.096555 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.096516 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" Nov 03 16:34:50.096785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.096574 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" event={"ID":"6bc977de-5e83-4494-a9aa-1214a2cd29e1","Type":"ContainerDied","Data":"fdf5c998ef82428f88d02491149d573d95c7cc37434e1ac866ca9cfaf30fbbc7"} Nov 03 16:34:50.096785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.096641 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w" event={"ID":"6bc977de-5e83-4494-a9aa-1214a2cd29e1","Type":"ContainerDied","Data":"8655399a0c7e5fe47a9dd1d5d8d65101e4e4d49267c3fe25c0ce571084e0e307"} Nov 03 16:34:50.096785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.096683 4947 scope.go:122] "RemoveContainer" containerID="fdf5c998ef82428f88d02491149d573d95c7cc37434e1ac866ca9cfaf30fbbc7" Nov 03 16:34:50.159426 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.159204 4947 scope.go:122] "RemoveContainer" containerID="c228478adf988b477f285ce01dc6f939fd3495e61884f6c0f236bf9100ce624e" Nov 03 16:34:50.171579 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.171553 4947 scope.go:122] "RemoveContainer" containerID="fdf5c998ef82428f88d02491149d573d95c7cc37434e1ac866ca9cfaf30fbbc7" Nov 03 16:34:50.171736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.171594 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w"] Nov 03 16:34:50.171939 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:34:50.171913 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdf5c998ef82428f88d02491149d573d95c7cc37434e1ac866ca9cfaf30fbbc7\": container with ID starting with fdf5c998ef82428f88d02491149d573d95c7cc37434e1ac866ca9cfaf30fbbc7 not found: ID does not exist" containerID="fdf5c998ef82428f88d02491149d573d95c7cc37434e1ac866ca9cfaf30fbbc7" Nov 03 16:34:50.172041 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.171949 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdf5c998ef82428f88d02491149d573d95c7cc37434e1ac866ca9cfaf30fbbc7"} err="failed to get container status \"fdf5c998ef82428f88d02491149d573d95c7cc37434e1ac866ca9cfaf30fbbc7\": rpc error: code = NotFound desc = could not find container \"fdf5c998ef82428f88d02491149d573d95c7cc37434e1ac866ca9cfaf30fbbc7\": container with ID starting with fdf5c998ef82428f88d02491149d573d95c7cc37434e1ac866ca9cfaf30fbbc7 not found: ID does not exist" Nov 03 16:34:50.172041 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.171972 4947 scope.go:122] "RemoveContainer" containerID="c228478adf988b477f285ce01dc6f939fd3495e61884f6c0f236bf9100ce624e" Nov 03 16:34:50.172228 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:34:50.172211 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c228478adf988b477f285ce01dc6f939fd3495e61884f6c0f236bf9100ce624e\": container with ID starting with c228478adf988b477f285ce01dc6f939fd3495e61884f6c0f236bf9100ce624e not found: ID does not exist" containerID="c228478adf988b477f285ce01dc6f939fd3495e61884f6c0f236bf9100ce624e" Nov 03 16:34:50.172287 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.172232 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c228478adf988b477f285ce01dc6f939fd3495e61884f6c0f236bf9100ce624e"} err="failed to get container status \"c228478adf988b477f285ce01dc6f939fd3495e61884f6c0f236bf9100ce624e\": rpc error: code = NotFound desc = could not find container \"c228478adf988b477f285ce01dc6f939fd3495e61884f6c0f236bf9100ce624e\": container with ID starting with c228478adf988b477f285ce01dc6f939fd3495e61884f6c0f236bf9100ce624e not found: ID does not exist" Nov 03 16:34:50.183251 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.183225 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c9fa1268-openshift-default-6655c8479c-dgz4w"] Nov 03 16:34:50.337229 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.337203 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:50.391879 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.391789 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df03b919-6262-4fd9-af88-788700865ebd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df03b919-6262-4fd9-af88-788700865ebd-istio-podinfo\") pod \"df03b919-6262-4fd9-af88-788700865ebd\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " Nov 03 16:34:50.391879 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.391873 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-workload-certs\") pod \"df03b919-6262-4fd9-af88-788700865ebd\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " Nov 03 16:34:50.392091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.391903 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df03b919-6262-4fd9-af88-788700865ebd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df03b919-6262-4fd9-af88-788700865ebd-istiod-ca-cert\") pod \"df03b919-6262-4fd9-af88-788700865ebd\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " Nov 03 16:34:50.392091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.391927 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-credential-socket\") pod \"df03b919-6262-4fd9-af88-788700865ebd\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " Nov 03 16:34:50.392091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.391966 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-istio-data\") pod \"df03b919-6262-4fd9-af88-788700865ebd\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " Nov 03 16:34:50.392091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.391991 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df03b919-6262-4fd9-af88-788700865ebd-istio-token\" (UniqueName: \"kubernetes.io/projected/df03b919-6262-4fd9-af88-788700865ebd-istio-token\") pod \"df03b919-6262-4fd9-af88-788700865ebd\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " Nov 03 16:34:50.392091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.392073 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-workload-socket\") pod \"df03b919-6262-4fd9-af88-788700865ebd\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " Nov 03 16:34:50.392340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.392125 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-istio-envoy\") pod \"df03b919-6262-4fd9-af88-788700865ebd\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " Nov 03 16:34:50.392340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.392208 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-workload-certs" pod "df03b919-6262-4fd9-af88-788700865ebd" (UID: "df03b919-6262-4fd9-af88-788700865ebd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:50.392340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.392233 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df03b919-6262-4fd9-af88-788700865ebd-kube-api-access-pcz68\" (UniqueName: \"kubernetes.io/projected/df03b919-6262-4fd9-af88-788700865ebd-kube-api-access-pcz68\") pod \"df03b919-6262-4fd9-af88-788700865ebd\" (UID: \"df03b919-6262-4fd9-af88-788700865ebd\") " Nov 03 16:34:50.392487 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.392354 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-istio-data" pod "df03b919-6262-4fd9-af88-788700865ebd" (UID: "df03b919-6262-4fd9-af88-788700865ebd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:50.392487 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.392397 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df03b919-6262-4fd9-af88-788700865ebd-istiod-ca-cert" pod "df03b919-6262-4fd9-af88-788700865ebd" (UID: "df03b919-6262-4fd9-af88-788700865ebd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:50.392613 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.392594 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-credential-socket" pod "df03b919-6262-4fd9-af88-788700865ebd" (UID: "df03b919-6262-4fd9-af88-788700865ebd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:50.392676 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.392635 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-workload-socket" pod "df03b919-6262-4fd9-af88-788700865ebd" (UID: "df03b919-6262-4fd9-af88-788700865ebd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:50.392783 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.392761 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:50.392783 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.392780 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df03b919-6262-4fd9-af88-788700865ebd-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:50.392897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.392793 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:50.392897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.392806 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:50.392897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.392821 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:50.394569 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.394545 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-istio-envoy" pod "df03b919-6262-4fd9-af88-788700865ebd" (UID: "df03b919-6262-4fd9-af88-788700865ebd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:34:50.394719 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.394672 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df03b919-6262-4fd9-af88-788700865ebd-istio-podinfo" pod "df03b919-6262-4fd9-af88-788700865ebd" (UID: "df03b919-6262-4fd9-af88-788700865ebd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:34:50.394989 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.394960 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df03b919-6262-4fd9-af88-788700865ebd-kube-api-access-pcz68" pod "df03b919-6262-4fd9-af88-788700865ebd" (UID: "df03b919-6262-4fd9-af88-788700865ebd"). InnerVolumeSpecName "kube-api-access-pcz68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:50.394989 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.394959 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df03b919-6262-4fd9-af88-788700865ebd-istio-token" pod "df03b919-6262-4fd9-af88-788700865ebd" (UID: "df03b919-6262-4fd9-af88-788700865ebd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:50.494597 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.494557 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcz68\" (UniqueName: \"kubernetes.io/projected/df03b919-6262-4fd9-af88-788700865ebd-kube-api-access-pcz68\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:50.494597 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.494590 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df03b919-6262-4fd9-af88-788700865ebd-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:50.494597 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.494600 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df03b919-6262-4fd9-af88-788700865ebd-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:50.494597 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.494610 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df03b919-6262-4fd9-af88-788700865ebd-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:50.565981 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:50.565946 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc977de-5e83-4494-a9aa-1214a2cd29e1" path="/var/lib/kubelet/pods/6bc977de-5e83-4494-a9aa-1214a2cd29e1/volumes" Nov 03 16:34:51.111155 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:51.111119 4947 generic.go:356] "Generic (PLEG): container finished" podID="df03b919-6262-4fd9-af88-788700865ebd" containerID="ca306f1148cd69a08d98e38b272b7e3c3a222beb3c5fcb2039aea5402fa24df4" exitCode=0 Nov 03 16:34:51.111605 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:51.111197 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" event={"ID":"df03b919-6262-4fd9-af88-788700865ebd","Type":"ContainerDied","Data":"ca306f1148cd69a08d98e38b272b7e3c3a222beb3c5fcb2039aea5402fa24df4"} Nov 03 16:34:51.111605 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:51.111280 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" event={"ID":"df03b919-6262-4fd9-af88-788700865ebd","Type":"ContainerDied","Data":"e9eb6db7118a69c799eef05dae3c724e2699d4b12d93794ccfa5275a707485f4"} Nov 03 16:34:51.111605 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:51.111204 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2" Nov 03 16:34:51.111605 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:51.111302 4947 scope.go:122] "RemoveContainer" containerID="ca306f1148cd69a08d98e38b272b7e3c3a222beb3c5fcb2039aea5402fa24df4" Nov 03 16:34:51.120727 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:51.120675 4947 scope.go:122] "RemoveContainer" containerID="f4ad9579bc3c016d75dc2032b2d60887c6cb8a76bbb54bd419bcacb95f0ed15e" Nov 03 16:34:51.130632 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:51.130609 4947 scope.go:122] "RemoveContainer" containerID="ca306f1148cd69a08d98e38b272b7e3c3a222beb3c5fcb2039aea5402fa24df4" Nov 03 16:34:51.130980 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:34:51.130959 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca306f1148cd69a08d98e38b272b7e3c3a222beb3c5fcb2039aea5402fa24df4\": container with ID starting with ca306f1148cd69a08d98e38b272b7e3c3a222beb3c5fcb2039aea5402fa24df4 not found: ID does not exist" containerID="ca306f1148cd69a08d98e38b272b7e3c3a222beb3c5fcb2039aea5402fa24df4" Nov 03 16:34:51.131051 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:51.130989 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca306f1148cd69a08d98e38b272b7e3c3a222beb3c5fcb2039aea5402fa24df4"} err="failed to get container status \"ca306f1148cd69a08d98e38b272b7e3c3a222beb3c5fcb2039aea5402fa24df4\": rpc error: code = NotFound desc = could not find container \"ca306f1148cd69a08d98e38b272b7e3c3a222beb3c5fcb2039aea5402fa24df4\": container with ID starting with ca306f1148cd69a08d98e38b272b7e3c3a222beb3c5fcb2039aea5402fa24df4 not found: ID does not exist" Nov 03 16:34:51.131051 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:51.131007 4947 scope.go:122] "RemoveContainer" containerID="f4ad9579bc3c016d75dc2032b2d60887c6cb8a76bbb54bd419bcacb95f0ed15e" Nov 03 16:34:51.131270 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:34:51.131243 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ad9579bc3c016d75dc2032b2d60887c6cb8a76bbb54bd419bcacb95f0ed15e\": container with ID starting with f4ad9579bc3c016d75dc2032b2d60887c6cb8a76bbb54bd419bcacb95f0ed15e not found: ID does not exist" containerID="f4ad9579bc3c016d75dc2032b2d60887c6cb8a76bbb54bd419bcacb95f0ed15e" Nov 03 16:34:51.131359 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:51.131277 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ad9579bc3c016d75dc2032b2d60887c6cb8a76bbb54bd419bcacb95f0ed15e"} err="failed to get container status \"f4ad9579bc3c016d75dc2032b2d60887c6cb8a76bbb54bd419bcacb95f0ed15e\": rpc error: code = NotFound desc = could not find container \"f4ad9579bc3c016d75dc2032b2d60887c6cb8a76bbb54bd419bcacb95f0ed15e\": container with ID starting with f4ad9579bc3c016d75dc2032b2d60887c6cb8a76bbb54bd419bcacb95f0ed15e not found: ID does not exist" Nov 03 16:34:51.166360 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:51.166322 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2"] Nov 03 16:34:51.179491 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:51.179457 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f0b922f4-openshift-default-58bfff88f5-kpnp2"] Nov 03 16:34:52.038861 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.038832 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57"] Nov 03 16:34:52.039235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.039223 4947 cpu_manager.go:414] "containerMap: removing container" podUID="2bdcb312-0f38-400e-a242-4c3d3f426179" containerName="payload-processing" Nov 03 16:34:52.039278 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.039238 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bdcb312-0f38-400e-a242-4c3d3f426179" containerName="payload-processing" Nov 03 16:34:52.039278 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.039255 4947 cpu_manager.go:414] "containerMap: removing container" podUID="6bc977de-5e83-4494-a9aa-1214a2cd29e1" containerName="istio-proxy" Nov 03 16:34:52.039278 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.039260 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc977de-5e83-4494-a9aa-1214a2cd29e1" containerName="istio-proxy" Nov 03 16:34:52.039278 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.039274 4947 cpu_manager.go:414] "containerMap: removing container" podUID="df03b919-6262-4fd9-af88-788700865ebd" containerName="istio-proxy" Nov 03 16:34:52.039278 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.039278 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df03b919-6262-4fd9-af88-788700865ebd" containerName="istio-proxy" Nov 03 16:34:52.039427 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.039290 4947 cpu_manager.go:414] "containerMap: removing container" podUID="6bc977de-5e83-4494-a9aa-1214a2cd29e1" containerName="istio-proxy" Nov 03 16:34:52.039427 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.039295 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc977de-5e83-4494-a9aa-1214a2cd29e1" containerName="istio-proxy" Nov 03 16:34:52.039427 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.039310 4947 cpu_manager.go:414] "containerMap: removing container" podUID="df03b919-6262-4fd9-af88-788700865ebd" containerName="istio-proxy" Nov 03 16:34:52.039427 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.039315 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df03b919-6262-4fd9-af88-788700865ebd" containerName="istio-proxy" Nov 03 16:34:52.039427 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.039376 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc977de-5e83-4494-a9aa-1214a2cd29e1" containerName="istio-proxy" Nov 03 16:34:52.039427 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.039385 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="df03b919-6262-4fd9-af88-788700865ebd" containerName="istio-proxy" Nov 03 16:34:52.039427 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.039392 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bdcb312-0f38-400e-a242-4c3d3f426179" containerName="payload-processing" Nov 03 16:34:52.039427 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.039399 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc977de-5e83-4494-a9aa-1214a2cd29e1" containerName="istio-proxy" Nov 03 16:34:52.042541 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.042524 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" Nov 03 16:34:52.071807 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.071776 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-7519b7-serving-cert\"" Nov 03 16:34:52.074815 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.074785 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57"] Nov 03 16:34:52.116238 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.116211 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39106eb-e15d-4413-9706-808f2de6d102-metrics-certs\") pod \"maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57\" (UID: \"a39106eb-e15d-4413-9706-808f2de6d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" Nov 03 16:34:52.116645 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.116315 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj4wv\" (UniqueName: \"kubernetes.io/projected/a39106eb-e15d-4413-9706-808f2de6d102-kube-api-access-wj4wv\") pod \"maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57\" (UID: \"a39106eb-e15d-4413-9706-808f2de6d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" Nov 03 16:34:52.116645 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.116425 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39106eb-e15d-4413-9706-808f2de6d102-maas-api-tls\") pod \"maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57\" (UID: \"a39106eb-e15d-4413-9706-808f2de6d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" Nov 03 16:34:52.116645 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.116471 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39106eb-e15d-4413-9706-808f2de6d102-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57\" (UID: \"a39106eb-e15d-4413-9706-808f2de6d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" Nov 03 16:34:52.226725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.225432 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39106eb-e15d-4413-9706-808f2de6d102-maas-api-tls\") pod \"maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57\" (UID: \"a39106eb-e15d-4413-9706-808f2de6d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" Nov 03 16:34:52.226725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.225497 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39106eb-e15d-4413-9706-808f2de6d102-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57\" (UID: \"a39106eb-e15d-4413-9706-808f2de6d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" Nov 03 16:34:52.226725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.225590 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39106eb-e15d-4413-9706-808f2de6d102-metrics-certs\") pod \"maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57\" (UID: \"a39106eb-e15d-4413-9706-808f2de6d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" Nov 03 16:34:52.226725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.225647 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj4wv\" (UniqueName: \"kubernetes.io/projected/a39106eb-e15d-4413-9706-808f2de6d102-kube-api-access-wj4wv\") pod \"maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57\" (UID: \"a39106eb-e15d-4413-9706-808f2de6d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" Nov 03 16:34:52.231079 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.230521 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39106eb-e15d-4413-9706-808f2de6d102-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57\" (UID: \"a39106eb-e15d-4413-9706-808f2de6d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" Nov 03 16:34:52.247630 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.246326 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39106eb-e15d-4413-9706-808f2de6d102-metrics-certs\") pod \"maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57\" (UID: \"a39106eb-e15d-4413-9706-808f2de6d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" Nov 03 16:34:52.252092 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.251112 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39106eb-e15d-4413-9706-808f2de6d102-maas-api-tls\") pod \"maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57\" (UID: \"a39106eb-e15d-4413-9706-808f2de6d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" Nov 03 16:34:52.252092 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.251837 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj4wv\" (UniqueName: \"kubernetes.io/projected/a39106eb-e15d-4413-9706-808f2de6d102-kube-api-access-wj4wv\") pod \"maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57\" (UID: \"a39106eb-e15d-4413-9706-808f2de6d102\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" Nov 03 16:34:52.354334 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.354247 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" Nov 03 16:34:52.564142 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.564105 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df03b919-6262-4fd9-af88-788700865ebd" path="/var/lib/kubelet/pods/df03b919-6262-4fd9-af88-788700865ebd/volumes" Nov 03 16:34:52.669617 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:52.669572 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57"] Nov 03 16:34:52.671893 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:34:52.671861 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39106eb_e15d_4413_9706_808f2de6d102.slice/crio-f618a4e834e09f1335727cd380e04db7ffc5f3c8cec17f68bbdc90d5aea08f2f WatchSource:0}: Error finding container f618a4e834e09f1335727cd380e04db7ffc5f3c8cec17f68bbdc90d5aea08f2f: Status 404 returned error can't find the container with id f618a4e834e09f1335727cd380e04db7ffc5f3c8cec17f68bbdc90d5aea08f2f Nov 03 16:34:53.124129 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:53.124098 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" event={"ID":"a39106eb-e15d-4413-9706-808f2de6d102","Type":"ContainerStarted","Data":"646c6d65368375dc78c59e6676512de7a33d690130c40ac10577da2df7c9df92"} Nov 03 16:34:53.124549 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:53.124148 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" event={"ID":"a39106eb-e15d-4413-9706-808f2de6d102","Type":"ContainerStarted","Data":"f618a4e834e09f1335727cd380e04db7ffc5f3c8cec17f68bbdc90d5aea08f2f"} Nov 03 16:34:53.124549 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:53.124207 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" Nov 03 16:34:53.174436 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:53.174379 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" podStartSLOduration=0.855483867 podStartE2EDuration="1.174362523s" totalImagesPullingTime="318.878656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:53.173133959 +0000 UTC m=+1236.762133928" watchObservedRunningTime="2026-11-03 16:34:53.174362523 +0000 UTC m=+1236.763362491" Nov 03 16:34:54.027990 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.027960 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8"] Nov 03 16:34:54.028457 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.028445 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="df03b919-6262-4fd9-af88-788700865ebd" containerName="istio-proxy" Nov 03 16:34:54.030649 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.030631 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 16:34:54.046971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.046939 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-3a2fa2-serving-cert\"" Nov 03 16:34:54.065746 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.062499 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8"] Nov 03 16:34:54.074413 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.074374 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8\" (UID: \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 16:34:54.074577 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.074494 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-metrics-certs\") pod \"maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8\" (UID: \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 16:34:54.074577 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.074535 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8\" (UID: \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 16:34:54.074748 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.074595 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tzml\" (UniqueName: \"kubernetes.io/projected/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-kube-api-access-8tzml\") pod \"maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8\" (UID: \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 16:34:54.175887 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.175849 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tzml\" (UniqueName: \"kubernetes.io/projected/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-kube-api-access-8tzml\") pod \"maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8\" (UID: \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 16:34:54.176413 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.175996 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8\" (UID: \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 16:34:54.176413 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.176171 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-metrics-certs\") pod \"maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8\" (UID: \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 16:34:54.176413 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.176218 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8\" (UID: \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 16:34:54.176413 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:34:54.176349 4947 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-serving-cert: secret "maas-api-e2e-worker-w3-3a2fa2-serving-cert" not found Nov 03 16:34:54.176413 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:34:54.176408 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-maas-api-tls podName:7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b nodeName:}" failed. No retries permitted until 2026-11-03 16:34:54.676392589 +0000 UTC m=+1238.265392536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-maas-api-tls") pod "maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" (UID: "7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b") : secret "maas-api-e2e-worker-w3-3a2fa2-serving-cert" not found Nov 03 16:34:54.176730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.176691 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8\" (UID: \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 16:34:54.178897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.178874 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-metrics-certs\") pod \"maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8\" (UID: \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 16:34:54.203660 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.203623 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tzml\" (UniqueName: \"kubernetes.io/projected/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-kube-api-access-8tzml\") pod \"maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8\" (UID: \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 16:34:54.682098 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.682059 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8\" (UID: \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 16:34:54.690604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.690497 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-maas-api-tls\") pod \"maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8\" (UID: \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 16:34:54.947463 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:54.947270 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 16:34:55.248393 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:55.248364 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8"] Nov 03 16:34:55.249392 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:34:55.249364 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f99db0f_b8dc_43cc_a2fd_0c3dcf5fe99b.slice/crio-1cf8477452758a7678e11f1dd053540b41b5edec71389db40c1db319ef6de676 WatchSource:0}: Error finding container 1cf8477452758a7678e11f1dd053540b41b5edec71389db40c1db319ef6de676: Status 404 returned error can't find the container with id 1cf8477452758a7678e11f1dd053540b41b5edec71389db40c1db319ef6de676 Nov 03 16:34:55.867953 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:55.865218 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:34:55.878146 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:55.878114 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q"] Nov 03 16:34:55.878369 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:55.878345 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" podUID="9edd4279-7a2a-4f62-a819-b5d9a9c1e848" containerName="maas-api" containerID="cri-o://f6bcfcfb60e4ec59d84f691da922f52d2ba8d94a7084fd2cffb3fa13cb85538d" gracePeriod=30 Nov 03 16:34:56.152146 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.151607 4947 generic.go:356] "Generic (PLEG): container finished" podID="9edd4279-7a2a-4f62-a819-b5d9a9c1e848" containerID="f6bcfcfb60e4ec59d84f691da922f52d2ba8d94a7084fd2cffb3fa13cb85538d" exitCode=0 Nov 03 16:34:56.152146 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.151743 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" event={"ID":"9edd4279-7a2a-4f62-a819-b5d9a9c1e848","Type":"ContainerDied","Data":"f6bcfcfb60e4ec59d84f691da922f52d2ba8d94a7084fd2cffb3fa13cb85538d"} Nov 03 16:34:56.154583 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.154239 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" event={"ID":"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b","Type":"ContainerStarted","Data":"49fb4eeeb4e85aa99e99c9a443c9aacffba382baba0a1eef04feba05fee186fe"} Nov 03 16:34:56.154583 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.154296 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" event={"ID":"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b","Type":"ContainerStarted","Data":"1cf8477452758a7678e11f1dd053540b41b5edec71389db40c1db319ef6de676"} Nov 03 16:34:56.154583 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.154537 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 16:34:56.206493 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.206439 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" podStartSLOduration=1.9173357850000001 podStartE2EDuration="2.206423182s" totalImagesPullingTime="289.087397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:56.204625581 +0000 UTC m=+1239.793625550" watchObservedRunningTime="2026-11-03 16:34:56.206423182 +0000 UTC m=+1239.795423150" Nov 03 16:34:56.230749 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.230724 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:56.318734 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.318670 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-metrics-certs\") pod \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\" (UID: \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\") " Nov 03 16:34:56.319216 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.318759 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-openshift-service-ca\") pod \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\" (UID: \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\") " Nov 03 16:34:56.319216 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.318848 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-maas-api-tls\") pod \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\" (UID: \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\") " Nov 03 16:34:56.319216 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.318880 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-kube-api-access-rbh5t\" (UniqueName: \"kubernetes.io/projected/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-kube-api-access-rbh5t\") pod \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\" (UID: \"9edd4279-7a2a-4f62-a819-b5d9a9c1e848\") " Nov 03 16:34:56.319216 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.319170 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-openshift-service-ca" pod "9edd4279-7a2a-4f62-a819-b5d9a9c1e848" (UID: "9edd4279-7a2a-4f62-a819-b5d9a9c1e848"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:34:56.321183 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.321155 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-maas-api-tls" pod "9edd4279-7a2a-4f62-a819-b5d9a9c1e848" (UID: "9edd4279-7a2a-4f62-a819-b5d9a9c1e848"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:34:56.321459 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.321438 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-metrics-certs" pod "9edd4279-7a2a-4f62-a819-b5d9a9c1e848" (UID: "9edd4279-7a2a-4f62-a819-b5d9a9c1e848"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:34:56.324565 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.324537 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-kube-api-access-rbh5t" pod "9edd4279-7a2a-4f62-a819-b5d9a9c1e848" (UID: "9edd4279-7a2a-4f62-a819-b5d9a9c1e848"). InnerVolumeSpecName "kube-api-access-rbh5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:34:56.426777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.422684 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:56.426777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.422742 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:56.426777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.422758 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbh5t\" (UniqueName: \"kubernetes.io/projected/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-kube-api-access-rbh5t\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:56.426777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.422771 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9edd4279-7a2a-4f62-a819-b5d9a9c1e848-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:34:56.656463 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.656429 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w"] Nov 03 16:34:56.656903 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.656889 4947 cpu_manager.go:414] "containerMap: removing container" podUID="9edd4279-7a2a-4f62-a819-b5d9a9c1e848" containerName="maas-api" Nov 03 16:34:56.656945 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.656905 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9edd4279-7a2a-4f62-a819-b5d9a9c1e848" containerName="maas-api" Nov 03 16:34:56.657007 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.656996 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9edd4279-7a2a-4f62-a819-b5d9a9c1e848" containerName="maas-api" Nov 03 16:34:56.659302 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.659279 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" Nov 03 16:34:56.678673 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.678593 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-460104-serving-cert\"" Nov 03 16:34:56.689218 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.689186 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w"] Nov 03 16:34:56.725007 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.724973 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a18780-ccc0-4d02-a76b-df4593a94e29-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w\" (UID: \"a6a18780-ccc0-4d02-a76b-df4593a94e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" Nov 03 16:34:56.725007 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.725009 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4c4p\" (UniqueName: \"kubernetes.io/projected/a6a18780-ccc0-4d02-a76b-df4593a94e29-kube-api-access-n4c4p\") pod \"maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w\" (UID: \"a6a18780-ccc0-4d02-a76b-df4593a94e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" Nov 03 16:34:56.725236 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.725042 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a18780-ccc0-4d02-a76b-df4593a94e29-maas-api-tls\") pod \"maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w\" (UID: \"a6a18780-ccc0-4d02-a76b-df4593a94e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" Nov 03 16:34:56.725236 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.725158 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6a18780-ccc0-4d02-a76b-df4593a94e29-metrics-certs\") pod \"maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w\" (UID: \"a6a18780-ccc0-4d02-a76b-df4593a94e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" Nov 03 16:34:56.825743 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.825691 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a18780-ccc0-4d02-a76b-df4593a94e29-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w\" (UID: \"a6a18780-ccc0-4d02-a76b-df4593a94e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" Nov 03 16:34:56.825743 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.825748 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4c4p\" (UniqueName: \"kubernetes.io/projected/a6a18780-ccc0-4d02-a76b-df4593a94e29-kube-api-access-n4c4p\") pod \"maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w\" (UID: \"a6a18780-ccc0-4d02-a76b-df4593a94e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" Nov 03 16:34:56.825974 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.825787 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a18780-ccc0-4d02-a76b-df4593a94e29-maas-api-tls\") pod \"maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w\" (UID: \"a6a18780-ccc0-4d02-a76b-df4593a94e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" Nov 03 16:34:56.825974 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.825813 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6a18780-ccc0-4d02-a76b-df4593a94e29-metrics-certs\") pod \"maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w\" (UID: \"a6a18780-ccc0-4d02-a76b-df4593a94e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" Nov 03 16:34:56.826360 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.826337 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a18780-ccc0-4d02-a76b-df4593a94e29-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w\" (UID: \"a6a18780-ccc0-4d02-a76b-df4593a94e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" Nov 03 16:34:56.828413 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.828390 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6a18780-ccc0-4d02-a76b-df4593a94e29-metrics-certs\") pod \"maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w\" (UID: \"a6a18780-ccc0-4d02-a76b-df4593a94e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" Nov 03 16:34:56.828591 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.828420 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a18780-ccc0-4d02-a76b-df4593a94e29-maas-api-tls\") pod \"maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w\" (UID: \"a6a18780-ccc0-4d02-a76b-df4593a94e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" Nov 03 16:34:56.850545 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.850509 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4c4p\" (UniqueName: \"kubernetes.io/projected/a6a18780-ccc0-4d02-a76b-df4593a94e29-kube-api-access-n4c4p\") pod \"maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w\" (UID: \"a6a18780-ccc0-4d02-a76b-df4593a94e29\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" Nov 03 16:34:56.973763 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:56.973656 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" Nov 03 16:34:57.160119 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:57.160090 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" Nov 03 16:34:57.160317 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:57.160128 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q" event={"ID":"9edd4279-7a2a-4f62-a819-b5d9a9c1e848","Type":"ContainerDied","Data":"e18edf7fd081060f578afcd9d4871e478c1d60bec56defdbcab5fb7c3e52826a"} Nov 03 16:34:57.160317 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:57.160189 4947 scope.go:122] "RemoveContainer" containerID="f6bcfcfb60e4ec59d84f691da922f52d2ba8d94a7084fd2cffb3fa13cb85538d" Nov 03 16:34:57.226039 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:57.225951 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q"] Nov 03 16:34:57.237593 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:57.237547 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-4d5492de-59599dbddc-4pf4q"] Nov 03 16:34:57.395774 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:57.395715 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w"] Nov 03 16:34:58.249845 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:58.249808 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" event={"ID":"a6a18780-ccc0-4d02-a76b-df4593a94e29","Type":"ContainerStarted","Data":"8a50d5fb09d1f79d7d86560a772b19513fc3483d684169c4f43ffa600d44fd74"} Nov 03 16:34:58.250211 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:58.250186 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" event={"ID":"a6a18780-ccc0-4d02-a76b-df4593a94e29","Type":"ContainerStarted","Data":"5dde0ab13e1c25726cb939bef3a86ab8073f651bacfe7dfb04d6fea5876a223f"} Nov 03 16:34:58.250355 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:58.250343 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" Nov 03 16:34:58.297773 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:58.297688 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" podStartSLOduration=1.865099915 podStartE2EDuration="2.297674579s" totalImagesPullingTime="432.574664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:34:58.296229484 +0000 UTC m=+1241.885229453" watchObservedRunningTime="2026-11-03 16:34:58.297674579 +0000 UTC m=+1241.886674548" Nov 03 16:34:58.564512 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:58.564436 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9edd4279-7a2a-4f62-a819-b5d9a9c1e848" path="/var/lib/kubelet/pods/9edd4279-7a2a-4f62-a819-b5d9a9c1e848/volumes" Nov 03 16:34:59.136096 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:34:59.136062 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" Nov 03 16:35:00.149811 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:35:00.149771 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:35:00.778045 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:00.778004 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:35:02.176853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:02.176821 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 16:35:04.134880 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:04.134835 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:35:04.135522 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:04.135486 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" podUID="79da839f-f698-42eb-9275-9999cb9a1306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:35:04.261026 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:04.260989 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" Nov 03 16:35:06.378295 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.378267 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn"] Nov 03 16:35:06.381462 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.381443 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn" Nov 03 16:35:06.428170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.423015 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-491905\"" Nov 03 16:35:06.435980 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.435946 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-491905-dockercfg-8flzm\"" Nov 03 16:35:06.463853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.463786 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn"] Nov 03 16:35:06.484722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.479285 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l"] Nov 03 16:35:06.563484 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.563441 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ddf118-a651-4201-85a6-e11ed761cceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn\" (UID: \"23ddf118-a651-4201-85a6-e11ed761cceb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn" Nov 03 16:35:06.565527 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.563936 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-554t5\" (UniqueName: \"kubernetes.io/projected/23ddf118-a651-4201-85a6-e11ed761cceb-kube-api-access-554t5\") pod \"payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn\" (UID: \"23ddf118-a651-4201-85a6-e11ed761cceb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn" Nov 03 16:35:06.593740 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.593633 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:35:06.594748 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.593760 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l" Nov 03 16:35:06.663375 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.663284 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l"] Nov 03 16:35:06.666437 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.666408 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ddf118-a651-4201-85a6-e11ed761cceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn\" (UID: \"23ddf118-a651-4201-85a6-e11ed761cceb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn" Nov 03 16:35:06.666613 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.666496 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-554t5\" (UniqueName: \"kubernetes.io/projected/23ddf118-a651-4201-85a6-e11ed761cceb-kube-api-access-554t5\") pod \"payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn\" (UID: \"23ddf118-a651-4201-85a6-e11ed761cceb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn" Nov 03 16:35:06.666613 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.666569 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkctf\" (UniqueName: \"kubernetes.io/projected/182c2154-67b3-47fe-a0df-730274bde9e0-kube-api-access-bkctf\") pod \"payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l\" (UID: \"182c2154-67b3-47fe-a0df-730274bde9e0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l" Nov 03 16:35:06.666770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.666651 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/182c2154-67b3-47fe-a0df-730274bde9e0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l\" (UID: \"182c2154-67b3-47fe-a0df-730274bde9e0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l" Nov 03 16:35:06.667381 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.667354 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ddf118-a651-4201-85a6-e11ed761cceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn\" (UID: \"23ddf118-a651-4201-85a6-e11ed761cceb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn" Nov 03 16:35:06.692752 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.692681 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-554t5\" (UniqueName: \"kubernetes.io/projected/23ddf118-a651-4201-85a6-e11ed761cceb-kube-api-access-554t5\") pod \"payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn\" (UID: \"23ddf118-a651-4201-85a6-e11ed761cceb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn" Nov 03 16:35:06.696918 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.696881 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn" Nov 03 16:35:06.786753 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.780938 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/182c2154-67b3-47fe-a0df-730274bde9e0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l\" (UID: \"182c2154-67b3-47fe-a0df-730274bde9e0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l" Nov 03 16:35:06.786753 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.781120 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkctf\" (UniqueName: \"kubernetes.io/projected/182c2154-67b3-47fe-a0df-730274bde9e0-kube-api-access-bkctf\") pod \"payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l\" (UID: \"182c2154-67b3-47fe-a0df-730274bde9e0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l" Nov 03 16:35:06.790567 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.790503 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/182c2154-67b3-47fe-a0df-730274bde9e0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l\" (UID: \"182c2154-67b3-47fe-a0df-730274bde9e0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l" Nov 03 16:35:06.845755 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.841399 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkctf\" (UniqueName: \"kubernetes.io/projected/182c2154-67b3-47fe-a0df-730274bde9e0-kube-api-access-bkctf\") pod \"payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l\" (UID: \"182c2154-67b3-47fe-a0df-730274bde9e0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l" Nov 03 16:35:06.977067 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:06.976801 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l" Nov 03 16:35:07.183849 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:07.182689 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn"] Nov 03 16:35:07.184660 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:35:07.184622 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ddf118_a651_4201_85a6_e11ed761cceb.slice/crio-1e848cbf1783a9057bfc329dc0d93091bfa34b09a8f009023a35df8f0396dba9 WatchSource:0}: Error finding container 1e848cbf1783a9057bfc329dc0d93091bfa34b09a8f009023a35df8f0396dba9: Status 404 returned error can't find the container with id 1e848cbf1783a9057bfc329dc0d93091bfa34b09a8f009023a35df8f0396dba9 Nov 03 16:35:07.412898 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:07.412849 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn" event={"ID":"23ddf118-a651-4201-85a6-e11ed761cceb","Type":"ContainerStarted","Data":"f49181cadc635fb61be9c711bfbae99b5a6510d8e59454b9550076a717f7c164"} Nov 03 16:35:07.413370 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:07.412953 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn" event={"ID":"23ddf118-a651-4201-85a6-e11ed761cceb","Type":"ContainerStarted","Data":"1e848cbf1783a9057bfc329dc0d93091bfa34b09a8f009023a35df8f0396dba9"} Nov 03 16:35:07.413370 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:07.413098 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn" Nov 03 16:35:07.420837 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:07.420805 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l"] Nov 03 16:35:07.421482 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:35:07.421455 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182c2154_67b3_47fe_a0df_730274bde9e0.slice/crio-f00d58c1e69383612952b30daa88d98f4ff3bf9546fe6262f071d6c992140cad WatchSource:0}: Error finding container f00d58c1e69383612952b30daa88d98f4ff3bf9546fe6262f071d6c992140cad: Status 404 returned error can't find the container with id f00d58c1e69383612952b30daa88d98f4ff3bf9546fe6262f071d6c992140cad Nov 03 16:35:07.457725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:07.457657 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn" podStartSLOduration=1.457642372 podStartE2EDuration="1.457642372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:07.456474811 +0000 UTC m=+1251.045474806" watchObservedRunningTime="2026-11-03 16:35:07.457642372 +0000 UTC m=+1251.046642335" Nov 03 16:35:08.422379 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:08.422327 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l" event={"ID":"182c2154-67b3-47fe-a0df-730274bde9e0","Type":"ContainerStarted","Data":"7bf333998bc1f8d671d7397c93cf29a5d0141a537dde886fb2d0a65ee3b029f3"} Nov 03 16:35:08.422880 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:08.422398 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l" event={"ID":"182c2154-67b3-47fe-a0df-730274bde9e0","Type":"ContainerStarted","Data":"f00d58c1e69383612952b30daa88d98f4ff3bf9546fe6262f071d6c992140cad"} Nov 03 16:35:08.422880 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:08.422750 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l" Nov 03 16:35:08.466877 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:08.466805 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l" podStartSLOduration=2.466786237 podStartE2EDuration="2.466786237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:08.466416143 +0000 UTC m=+1252.055416114" watchObservedRunningTime="2026-11-03 16:35:08.466786237 +0000 UTC m=+1252.055786210" Nov 03 16:35:09.430380 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:09.430346 4947 generic.go:356] "Generic (PLEG): container finished" podID="79da839f-f698-42eb-9275-9999cb9a1306" containerID="dfd75461b6ce7d634582b29fe24a6e8fa4f494a0780227ecd729038ccaa03a2f" exitCode=0 Nov 03 16:35:09.430917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:09.430425 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" event={"ID":"79da839f-f698-42eb-9275-9999cb9a1306","Type":"ContainerDied","Data":"dfd75461b6ce7d634582b29fe24a6e8fa4f494a0780227ecd729038ccaa03a2f"} Nov 03 16:35:09.431065 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:09.431049 4947 scope.go:122] "RemoveContainer" containerID="dfd75461b6ce7d634582b29fe24a6e8fa4f494a0780227ecd729038ccaa03a2f" Nov 03 16:35:09.432447 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:09.432412 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:35:09.432536 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:09.432503 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:35:09.432590 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:09.432551 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:35:10.449006 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:10.448956 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" event={"ID":"79da839f-f698-42eb-9275-9999cb9a1306","Type":"ContainerStarted","Data":"8b46d09daed868b7146f86e8676e14402d88717f73559001be08cc664448a784"} Nov 03 16:35:10.647969 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:10.647933 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:35:10.648208 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:10.648134 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 16:35:10.648208 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:10.648178 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" podUID="79da839f-f698-42eb-9275-9999cb9a1306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 16:35:11.219430 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:11.219388 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:35:11.465588 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:11.465496 4947 generic.go:356] "Generic (PLEG): container finished" podID="adc797dd-c98b-4af6-8939-770401241bf0" containerID="bb75213e9c1538e13c87667b0d6e2801ce0f1d4c37630a6013cf489cd90caf0d" exitCode=0 Nov 03 16:35:11.467462 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:11.466413 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" event={"ID":"adc797dd-c98b-4af6-8939-770401241bf0","Type":"ContainerDied","Data":"bb75213e9c1538e13c87667b0d6e2801ce0f1d4c37630a6013cf489cd90caf0d"} Nov 03 16:35:11.467462 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:11.466818 4947 scope.go:122] "RemoveContainer" containerID="bb75213e9c1538e13c87667b0d6e2801ce0f1d4c37630a6013cf489cd90caf0d" Nov 03 16:35:11.468378 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:11.468343 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:35:11.468469 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:11.468431 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:35:11.468506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:11.468476 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:35:11.648233 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:11.648201 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 16:35:11.648451 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:11.648269 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" podUID="79da839f-f698-42eb-9275-9999cb9a1306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 16:35:12.475529 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:12.475194 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" event={"ID":"adc797dd-c98b-4af6-8939-770401241bf0","Type":"ContainerStarted","Data":"c9bb15903a12cd75d6e3d2abd0c2ddd427a4bf156735c531f92a2dfa3335bc54"} Nov 03 16:35:12.477278 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:12.477247 4947 generic.go:356] "Generic (PLEG): container finished" podID="8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" containerID="1bd0a0d962da6a0aa8a1b0b5ddd7aea25febefee980fe34c58319351bc291dcb" exitCode=0 Nov 03 16:35:12.477402 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:12.477315 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" event={"ID":"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20","Type":"ContainerDied","Data":"1bd0a0d962da6a0aa8a1b0b5ddd7aea25febefee980fe34c58319351bc291dcb"} Nov 03 16:35:12.477627 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:12.477613 4947 scope.go:122] "RemoveContainer" containerID="1bd0a0d962da6a0aa8a1b0b5ddd7aea25febefee980fe34c58319351bc291dcb" Nov 03 16:35:12.478892 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:12.478863 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:35:12.478943 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:12.478932 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:35:12.478981 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:12.478971 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:35:12.658585 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:12.657761 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:35:12.658585 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:12.657880 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 16:35:12.658585 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:12.657920 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" podUID="adc797dd-c98b-4af6-8939-770401241bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 16:35:12.658585 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:12.658365 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 16:35:12.658585 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:12.658398 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" podUID="79da839f-f698-42eb-9275-9999cb9a1306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 16:35:12.978560 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:12.978465 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579"] Nov 03 16:35:12.982031 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:12.982010 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" Nov 03 16:35:13.018605 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.018539 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-491905-serving-cert\"" Nov 03 16:35:13.040963 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.040852 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579"] Nov 03 16:35:13.082251 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.079055 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2574a131-2be5-4179-9cb2-71980b0feeb8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-491905-6c57979cd4-2l579\" (UID: \"2574a131-2be5-4179-9cb2-71980b0feeb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" Nov 03 16:35:13.082251 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.079099 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pntn2\" (UniqueName: \"kubernetes.io/projected/2574a131-2be5-4179-9cb2-71980b0feeb8-kube-api-access-pntn2\") pod \"maas-api-e2e-worker-w1-491905-6c57979cd4-2l579\" (UID: \"2574a131-2be5-4179-9cb2-71980b0feeb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" Nov 03 16:35:13.082251 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.079117 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2574a131-2be5-4179-9cb2-71980b0feeb8-metrics-certs\") pod \"maas-api-e2e-worker-w1-491905-6c57979cd4-2l579\" (UID: \"2574a131-2be5-4179-9cb2-71980b0feeb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" Nov 03 16:35:13.082251 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.079232 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2574a131-2be5-4179-9cb2-71980b0feeb8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-491905-6c57979cd4-2l579\" (UID: \"2574a131-2be5-4179-9cb2-71980b0feeb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" Nov 03 16:35:13.180771 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.180657 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2574a131-2be5-4179-9cb2-71980b0feeb8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-491905-6c57979cd4-2l579\" (UID: \"2574a131-2be5-4179-9cb2-71980b0feeb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" Nov 03 16:35:13.180936 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.180809 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2574a131-2be5-4179-9cb2-71980b0feeb8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-491905-6c57979cd4-2l579\" (UID: \"2574a131-2be5-4179-9cb2-71980b0feeb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" Nov 03 16:35:13.180936 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.180830 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pntn2\" (UniqueName: \"kubernetes.io/projected/2574a131-2be5-4179-9cb2-71980b0feeb8-kube-api-access-pntn2\") pod \"maas-api-e2e-worker-w1-491905-6c57979cd4-2l579\" (UID: \"2574a131-2be5-4179-9cb2-71980b0feeb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" Nov 03 16:35:13.180936 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.180847 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2574a131-2be5-4179-9cb2-71980b0feeb8-metrics-certs\") pod \"maas-api-e2e-worker-w1-491905-6c57979cd4-2l579\" (UID: \"2574a131-2be5-4179-9cb2-71980b0feeb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" Nov 03 16:35:13.181577 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.181552 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2574a131-2be5-4179-9cb2-71980b0feeb8-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-491905-6c57979cd4-2l579\" (UID: \"2574a131-2be5-4179-9cb2-71980b0feeb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" Nov 03 16:35:13.183514 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.183490 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2574a131-2be5-4179-9cb2-71980b0feeb8-maas-api-tls\") pod \"maas-api-e2e-worker-w1-491905-6c57979cd4-2l579\" (UID: \"2574a131-2be5-4179-9cb2-71980b0feeb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" Nov 03 16:35:13.183844 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.183826 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2574a131-2be5-4179-9cb2-71980b0feeb8-metrics-certs\") pod \"maas-api-e2e-worker-w1-491905-6c57979cd4-2l579\" (UID: \"2574a131-2be5-4179-9cb2-71980b0feeb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" Nov 03 16:35:13.255856 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.251442 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pntn2\" (UniqueName: \"kubernetes.io/projected/2574a131-2be5-4179-9cb2-71980b0feeb8-kube-api-access-pntn2\") pod \"maas-api-e2e-worker-w1-491905-6c57979cd4-2l579\" (UID: \"2574a131-2be5-4179-9cb2-71980b0feeb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" Nov 03 16:35:13.314263 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.314232 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" Nov 03 16:35:13.493746 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.493709 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" event={"ID":"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20","Type":"ContainerStarted","Data":"3fd441d421286b8c7998817076db3baea2526f99df4373c702b3c915fd87d411"} Nov 03 16:35:13.681741 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.679198 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 16:35:13.681741 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.679257 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" podUID="adc797dd-c98b-4af6-8939-770401241bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 16:35:13.681741 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.679940 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 16:35:13.681741 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.679978 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" podUID="79da839f-f698-42eb-9275-9999cb9a1306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 16:35:13.875629 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.862257 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:35:13.875629 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.865900 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 16:35:13.875629 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.865969 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" podUID="8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 16:35:13.877755 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:13.877690 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579"] Nov 03 16:35:14.556723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:14.552774 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" event={"ID":"2574a131-2be5-4179-9cb2-71980b0feeb8","Type":"ContainerStarted","Data":"b17bdef4c1d9ba7982dcf341961975378918a481bdbfe47a6c4582cfb49230a6"} Nov 03 16:35:14.658467 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:14.658363 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 16:35:14.658784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:14.658751 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" podUID="79da839f-f698-42eb-9275-9999cb9a1306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 16:35:14.659038 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:14.659021 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 16:35:14.659209 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:14.659188 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" podUID="adc797dd-c98b-4af6-8939-770401241bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 16:35:14.833137 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:14.833099 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 16:35:14.833414 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:14.833387 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" podUID="8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 16:35:15.581280 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:15.581243 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" event={"ID":"2574a131-2be5-4179-9cb2-71980b0feeb8","Type":"ContainerStarted","Data":"22a4f66c66af7a0019532c8856a12793c341c10ba1ac6526f8608b3ee67f4196"} Nov 03 16:35:15.581734 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:15.581488 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" Nov 03 16:35:15.641864 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:15.641793 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" podStartSLOduration=3.131160758 podStartE2EDuration="3.6417717s" totalImagesPullingTime="510.610942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:15.64020281 +0000 UTC m=+1259.229202782" watchObservedRunningTime="2026-11-03 16:35:15.6417717 +0000 UTC m=+1259.230771670" Nov 03 16:35:15.648561 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:15.648526 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Nov 03 16:35:15.648918 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:15.648873 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" podUID="79da839f-f698-42eb-9275-9999cb9a1306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Nov 03 16:35:15.649826 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:15.649794 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 16:35:15.653720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:15.649973 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" podUID="adc797dd-c98b-4af6-8939-770401241bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 16:35:15.837785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:15.837116 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 16:35:15.837785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:15.837179 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" podUID="8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 16:35:16.666069 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:16.666032 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 16:35:16.666724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:16.666651 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" podUID="adc797dd-c98b-4af6-8939-770401241bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 16:35:16.833724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:16.833092 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 16:35:16.833724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:16.833153 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" podUID="8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 16:35:16.862937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:16.841024 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:35:16.862937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:16.842444 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:35:16.862937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:16.862309 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:35:17.075198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:17.075161 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:35:17.645738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:17.645106 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Nov 03 16:35:17.645738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:17.645171 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" podUID="adc797dd-c98b-4af6-8939-770401241bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Nov 03 16:35:17.833133 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:17.833092 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 16:35:17.833550 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:17.833152 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" podUID="8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 16:35:18.425913 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:18.425882 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn" Nov 03 16:35:18.649658 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:18.649621 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:35:18.650034 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:18.650016 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:35:18.650874 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:18.650854 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 16:35:18.833117 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:18.833079 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Nov 03 16:35:18.833302 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:18.833142 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" podUID="8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Nov 03 16:35:19.434398 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:19.434367 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l" Nov 03 16:35:19.836942 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:19.836913 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:35:19.838050 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:19.838033 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:35:20.592913 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:20.592879 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:35:21.593918 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:21.593889 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" Nov 03 16:35:27.191871 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:27.191827 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:35:28.238347 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.238315 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4"] Nov 03 16:35:28.242156 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.242120 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.251588 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.251564 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-tls-certs\") pod \"simulator-w0-71d7b4-kserve-cb95b595d-dwcd4\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.251735 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.251606 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-dshm\") pod \"simulator-w0-71d7b4-kserve-cb95b595d-dwcd4\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.251735 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.251630 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qphgd\" (UniqueName: \"kubernetes.io/projected/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-kube-api-access-qphgd\") pod \"simulator-w0-71d7b4-kserve-cb95b595d-dwcd4\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.251858 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.251782 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-tmp-dir\") pod \"simulator-w0-71d7b4-kserve-cb95b595d-dwcd4\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.251858 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.251823 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-model-cache\") pod \"simulator-w0-71d7b4-kserve-cb95b595d-dwcd4\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.251933 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.251917 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-home\") pod \"simulator-w0-71d7b4-kserve-cb95b595d-dwcd4\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.258930 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.258909 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-71d7b4\"/\"kube-root-ca.crt\"" Nov 03 16:35:28.271987 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.271954 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-71d7b4\"/\"openshift-service-ca.crt\"" Nov 03 16:35:28.272137 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.272047 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-71d7b4\"/\"default-dockercfg-h4hqv\"" Nov 03 16:35:28.272336 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.272314 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-71d7b4\"/\"simulator-w0-71d7b4-kserve-self-signed-certs\"" Nov 03 16:35:28.274601 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.274577 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4"] Nov 03 16:35:28.352447 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.352412 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-tls-certs\") pod \"simulator-w0-71d7b4-kserve-cb95b595d-dwcd4\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.352640 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.352461 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-dshm\") pod \"simulator-w0-71d7b4-kserve-cb95b595d-dwcd4\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.352640 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.352487 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qphgd\" (UniqueName: \"kubernetes.io/projected/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-kube-api-access-qphgd\") pod \"simulator-w0-71d7b4-kserve-cb95b595d-dwcd4\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.352640 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.352532 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-tmp-dir\") pod \"simulator-w0-71d7b4-kserve-cb95b595d-dwcd4\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.352640 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.352549 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-model-cache\") pod \"simulator-w0-71d7b4-kserve-cb95b595d-dwcd4\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.352640 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.352582 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-home\") pod \"simulator-w0-71d7b4-kserve-cb95b595d-dwcd4\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.353056 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.353035 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-tmp-dir\") pod \"simulator-w0-71d7b4-kserve-cb95b595d-dwcd4\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.353122 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.353081 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-home\") pod \"simulator-w0-71d7b4-kserve-cb95b595d-dwcd4\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.353122 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.353114 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-model-cache\") pod \"simulator-w0-71d7b4-kserve-cb95b595d-dwcd4\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.355084 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.355067 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-dshm\") pod \"simulator-w0-71d7b4-kserve-cb95b595d-dwcd4\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.355183 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.355159 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-tls-certs\") pod \"simulator-w0-71d7b4-kserve-cb95b595d-dwcd4\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.377313 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.377285 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qphgd\" (UniqueName: \"kubernetes.io/projected/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-kube-api-access-qphgd\") pod \"simulator-w0-71d7b4-kserve-cb95b595d-dwcd4\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.566596 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.566564 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:28.871664 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:28.871633 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4"] Nov 03 16:35:28.872447 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:35:28.872424 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbe4b1e0_bc76_4e9b_93e9_f8a59d53c780.slice/crio-9cf6fd275beeb850a2568531834a85bf2286d6cd59b3511ad99647b5c17c9ace WatchSource:0}: Error finding container 9cf6fd275beeb850a2568531834a85bf2286d6cd59b3511ad99647b5c17c9ace: Status 404 returned error can't find the container with id 9cf6fd275beeb850a2568531834a85bf2286d6cd59b3511ad99647b5c17c9ace Nov 03 16:35:29.673763 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:29.673725 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" event={"ID":"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780","Type":"ContainerStarted","Data":"c2e571a588fd7326173d52cbdb513a675260c068d3199f9820a272cd07535d18"} Nov 03 16:35:29.674188 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:29.673774 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" event={"ID":"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780","Type":"ContainerStarted","Data":"9cf6fd275beeb850a2568531834a85bf2286d6cd59b3511ad99647b5c17c9ace"} Nov 03 16:35:29.727025 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:29.726944 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podStartSLOduration=1.726928705 podStartE2EDuration="1.726928705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:29.726752038 +0000 UTC m=+1273.315752007" watchObservedRunningTime="2026-11-03 16:35:29.726928705 +0000 UTC m=+1273.315928673" Nov 03 16:35:30.130830 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:35:30.130788 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:35:30.567343 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:30.567319 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:36.605749 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:36.605713 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:35:38.566818 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:38.566774 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:38.579348 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:38.579317 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:38.734190 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:38.734158 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:35:47.782736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.782676 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d"] Nov 03 16:35:47.787409 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.787385 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:47.817244 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.817212 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3a2fa2\"/\"openshift-service-ca.crt\"" Nov 03 16:35:47.817441 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.817294 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-3a2fa2\"/\"kube-root-ca.crt\"" Nov 03 16:35:47.817441 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.817409 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3a2fa2\"/\"premium-w3-3a2fa2-kserve-self-signed-certs\"" Nov 03 16:35:47.817541 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.817480 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-3a2fa2\"/\"default-dockercfg-zzp5s\"" Nov 03 16:35:47.820462 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.820434 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d"] Nov 03 16:35:47.893441 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.893389 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6e9f4a-d932-4a88-b640-6676db6c29f8-tls-certs\") pod \"premium-w3-3a2fa2-kserve-76bf569f78-pcq5d\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:47.893441 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.893425 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-home\") pod \"premium-w3-3a2fa2-kserve-76bf569f78-pcq5d\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:47.893738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.893451 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-tmp-dir\") pod \"premium-w3-3a2fa2-kserve-76bf569f78-pcq5d\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:47.893738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.893644 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25kwb\" (UniqueName: \"kubernetes.io/projected/aa6e9f4a-d932-4a88-b640-6676db6c29f8-kube-api-access-25kwb\") pod \"premium-w3-3a2fa2-kserve-76bf569f78-pcq5d\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:47.893738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.893676 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-dshm\") pod \"premium-w3-3a2fa2-kserve-76bf569f78-pcq5d\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:47.893738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.893722 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-model-cache\") pod \"premium-w3-3a2fa2-kserve-76bf569f78-pcq5d\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:47.995518 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.995475 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25kwb\" (UniqueName: \"kubernetes.io/projected/aa6e9f4a-d932-4a88-b640-6676db6c29f8-kube-api-access-25kwb\") pod \"premium-w3-3a2fa2-kserve-76bf569f78-pcq5d\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:47.996337 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.996301 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-dshm\") pod \"premium-w3-3a2fa2-kserve-76bf569f78-pcq5d\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:47.996522 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.996508 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-model-cache\") pod \"premium-w3-3a2fa2-kserve-76bf569f78-pcq5d\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:47.998176 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.998139 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6e9f4a-d932-4a88-b640-6676db6c29f8-tls-certs\") pod \"premium-w3-3a2fa2-kserve-76bf569f78-pcq5d\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:47.998351 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.998336 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-home\") pod \"premium-w3-3a2fa2-kserve-76bf569f78-pcq5d\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:47.998491 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.998478 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-tmp-dir\") pod \"premium-w3-3a2fa2-kserve-76bf569f78-pcq5d\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:48.000014 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:47.999992 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-tmp-dir\") pod \"premium-w3-3a2fa2-kserve-76bf569f78-pcq5d\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:48.003524 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:48.003493 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-home\") pod \"premium-w3-3a2fa2-kserve-76bf569f78-pcq5d\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:48.003859 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:48.003830 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-model-cache\") pod \"premium-w3-3a2fa2-kserve-76bf569f78-pcq5d\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:48.005905 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:48.005880 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-dshm\") pod \"premium-w3-3a2fa2-kserve-76bf569f78-pcq5d\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:48.008353 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:48.008316 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6e9f4a-d932-4a88-b640-6676db6c29f8-tls-certs\") pod \"premium-w3-3a2fa2-kserve-76bf569f78-pcq5d\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:48.020513 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:48.020485 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25kwb\" (UniqueName: \"kubernetes.io/projected/aa6e9f4a-d932-4a88-b640-6676db6c29f8-kube-api-access-25kwb\") pod \"premium-w3-3a2fa2-kserve-76bf569f78-pcq5d\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:48.101859 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:48.101767 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:48.160088 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:48.159824 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:35:48.409046 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:48.409016 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d"] Nov 03 16:35:48.409765 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:35:48.409736 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa6e9f4a_d932_4a88_b640_6676db6c29f8.slice/crio-b60982178bce997a5bc28db77a3203b8e792c8a86ca0b04a4bb9a9d5b526f30f WatchSource:0}: Error finding container b60982178bce997a5bc28db77a3203b8e792c8a86ca0b04a4bb9a9d5b526f30f: Status 404 returned error can't find the container with id b60982178bce997a5bc28db77a3203b8e792c8a86ca0b04a4bb9a9d5b526f30f Nov 03 16:35:48.781078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:48.781043 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" event={"ID":"aa6e9f4a-d932-4a88-b640-6676db6c29f8","Type":"ContainerStarted","Data":"01b816830c8070fa5291bed22a0475999fee19402ea5c8258bcf1684959d4b41"} Nov 03 16:35:48.781078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:48.781093 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" event={"ID":"aa6e9f4a-d932-4a88-b640-6676db6c29f8","Type":"ContainerStarted","Data":"b60982178bce997a5bc28db77a3203b8e792c8a86ca0b04a4bb9a9d5b526f30f"} Nov 03 16:35:48.830616 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:48.830555 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podStartSLOduration=1.8305396539999998 podStartE2EDuration="1.830539654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:35:48.826254304 +0000 UTC m=+1292.415254273" watchObservedRunningTime="2026-11-03 16:35:48.830539654 +0000 UTC m=+1292.419539636" Nov 03 16:35:49.102035 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:49.101952 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:58.102327 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:58.102240 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:58.115926 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:58.115896 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:35:58.863356 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:35:58.863323 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:36:00.130891 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:36:00.130857 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:36:00.180116 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:00.180083 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:36:14.372834 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:14.372795 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:36:15.467867 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.467825 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq"] Nov 03 16:36:15.471168 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.471147 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.485733 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.485670 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq"] Nov 03 16:36:15.496806 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.496775 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-491905\"/\"premium-w1-491905-kserve-self-signed-certs\"" Nov 03 16:36:15.496978 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.496855 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-491905\"/\"default-dockercfg-m22tm\"" Nov 03 16:36:15.500524 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.500499 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-491905\"/\"kube-root-ca.crt\"" Nov 03 16:36:15.500661 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.500622 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-491905\"/\"openshift-service-ca.crt\"" Nov 03 16:36:15.626169 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.626134 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-tmp-dir\") pod \"premium-w1-491905-kserve-6446897f6c-rs5sq\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.626354 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.626200 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-dshm\") pod \"premium-w1-491905-kserve-6446897f6c-rs5sq\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.626354 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.626290 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-model-cache\") pod \"premium-w1-491905-kserve-6446897f6c-rs5sq\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.626354 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.626319 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-tls-certs\") pod \"premium-w1-491905-kserve-6446897f6c-rs5sq\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.626464 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.626431 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qcm9\" (UniqueName: \"kubernetes.io/projected/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-kube-api-access-9qcm9\") pod \"premium-w1-491905-kserve-6446897f6c-rs5sq\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.626502 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.626469 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-home\") pod \"premium-w1-491905-kserve-6446897f6c-rs5sq\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.727849 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.727738 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-dshm\") pod \"premium-w1-491905-kserve-6446897f6c-rs5sq\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.727849 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.727821 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-model-cache\") pod \"premium-w1-491905-kserve-6446897f6c-rs5sq\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.728154 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.727856 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-tls-certs\") pod \"premium-w1-491905-kserve-6446897f6c-rs5sq\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.728154 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.727997 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qcm9\" (UniqueName: \"kubernetes.io/projected/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-kube-api-access-9qcm9\") pod \"premium-w1-491905-kserve-6446897f6c-rs5sq\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.728154 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.728073 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-home\") pod \"premium-w1-491905-kserve-6446897f6c-rs5sq\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.728154 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.728136 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-tmp-dir\") pod \"premium-w1-491905-kserve-6446897f6c-rs5sq\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.728815 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.728323 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-model-cache\") pod \"premium-w1-491905-kserve-6446897f6c-rs5sq\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.729175 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.729151 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-tmp-dir\") pod \"premium-w1-491905-kserve-6446897f6c-rs5sq\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.729341 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.729320 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-home\") pod \"premium-w1-491905-kserve-6446897f6c-rs5sq\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.730856 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.730826 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-dshm\") pod \"premium-w1-491905-kserve-6446897f6c-rs5sq\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.732476 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.732454 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-tls-certs\") pod \"premium-w1-491905-kserve-6446897f6c-rs5sq\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.756659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.756621 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qcm9\" (UniqueName: \"kubernetes.io/projected/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-kube-api-access-9qcm9\") pod \"premium-w1-491905-kserve-6446897f6c-rs5sq\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:15.784175 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:15.784137 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:16.089530 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:16.089505 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq"] Nov 03 16:36:16.091037 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:36:16.091005 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfaa05b2a_faca_4a3f_a3c7_c12c7ac6f12a.slice/crio-5c3666703ec6aa44ce8dd6949eaab7e63c31c9a72ee624b96a1afa7c5fe30411 WatchSource:0}: Error finding container 5c3666703ec6aa44ce8dd6949eaab7e63c31c9a72ee624b96a1afa7c5fe30411: Status 404 returned error can't find the container with id 5c3666703ec6aa44ce8dd6949eaab7e63c31c9a72ee624b96a1afa7c5fe30411 Nov 03 16:36:16.962759 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:16.962726 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" event={"ID":"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a","Type":"ContainerStarted","Data":"5c4ec8e254d836f4e5d8eb3149e8c2e53566da946789536f1ad3411969176d46"} Nov 03 16:36:16.962759 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:16.962766 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" event={"ID":"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a","Type":"ContainerStarted","Data":"5c3666703ec6aa44ce8dd6949eaab7e63c31c9a72ee624b96a1afa7c5fe30411"} Nov 03 16:36:17.015156 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:17.015094 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" podStartSLOduration=2.015079159 podStartE2EDuration="2.015079159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:36:17.013424986 +0000 UTC m=+1320.602424955" watchObservedRunningTime="2026-11-03 16:36:17.015079159 +0000 UTC m=+1320.604079128" Nov 03 16:36:17.784922 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:17.784883 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:25.784778 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:25.784734 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:25.797941 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:25.797905 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:26.026609 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:26.026579 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:36:30.130856 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:36:30.130819 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:36:38.966651 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:38.966613 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:36:43.938381 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:43.938343 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5"] Nov 03 16:36:43.941506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:43.941489 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:43.958126 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:43.958093 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-491905-openshift-default-dockercfg-dc6wb\"" Nov 03 16:36:43.975450 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:43.975417 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5"] Nov 03 16:36:44.087588 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.087555 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.087789 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.087594 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.087789 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.087666 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7fst\" (UniqueName: \"kubernetes.io/projected/8c8da48a-94af-4d09-aef7-21c1a69092c7-kube-api-access-r7fst\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.087875 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.087786 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8da48a-94af-4d09-aef7-21c1a69092c7-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.087875 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.087831 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.087939 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.087886 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.087939 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.087926 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.088006 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.087945 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.088038 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.088025 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.189149 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.189063 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.189293 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.189157 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.189293 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.189199 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.189293 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.189258 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7fst\" (UniqueName: \"kubernetes.io/projected/8c8da48a-94af-4d09-aef7-21c1a69092c7-kube-api-access-r7fst\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.189417 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.189311 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8da48a-94af-4d09-aef7-21c1a69092c7-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.189417 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.189361 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.189417 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.189400 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.189529 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.189465 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.189529 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.189491 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.189606 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.189534 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.189998 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.189974 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.190338 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.190118 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.190475 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.190356 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8da48a-94af-4d09-aef7-21c1a69092c7-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.190475 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.190412 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.192016 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.191989 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.192130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.192107 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.214332 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.214298 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.214651 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.214622 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7fst\" (UniqueName: \"kubernetes.io/projected/8c8da48a-94af-4d09-aef7-21c1a69092c7-kube-api-access-r7fst\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.254303 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.254258 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:44.583296 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:36:44.583269 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8da48a_94af_4d09_aef7_21c1a69092c7.slice/crio-f18ccf2411168df660c9def6433d26d233dfe76bded1a8c505cb6fda160067d7 WatchSource:0}: Error finding container f18ccf2411168df660c9def6433d26d233dfe76bded1a8c505cb6fda160067d7: Status 404 returned error can't find the container with id f18ccf2411168df660c9def6433d26d233dfe76bded1a8c505cb6fda160067d7 Nov 03 16:36:44.584329 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.584304 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5"] Nov 03 16:36:44.585533 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.585503 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:36:44.585591 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.585566 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:36:44.585630 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:44.585595 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:36:45.119777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:45.119743 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" event={"ID":"8c8da48a-94af-4d09-aef7-21c1a69092c7","Type":"ContainerStarted","Data":"4e7780711b3da942e5e1abad8f7b2e0a3bb094e22119043e2282a9398ed212ea"} Nov 03 16:36:45.120244 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:45.119797 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" event={"ID":"8c8da48a-94af-4d09-aef7-21c1a69092c7","Type":"ContainerStarted","Data":"f18ccf2411168df660c9def6433d26d233dfe76bded1a8c505cb6fda160067d7"} Nov 03 16:36:45.196637 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:45.196589 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" podStartSLOduration=2.196574504 podStartE2EDuration="2.196574504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:36:45.182877579 +0000 UTC m=+1348.771877538" watchObservedRunningTime="2026-11-03 16:36:45.196574504 +0000 UTC m=+1348.785574470" Nov 03 16:36:45.255411 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:45.255373 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:45.257119 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:45.257083 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:36:45.257258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:45.257159 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" podUID="8c8da48a-94af-4d09-aef7-21c1a69092c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 16:36:46.255399 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:46.255366 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:36:46.255812 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:46.255425 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" podUID="8c8da48a-94af-4d09-aef7-21c1a69092c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 16:36:47.255218 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:47.255185 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:36:47.255419 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:47.255244 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" podUID="8c8da48a-94af-4d09-aef7-21c1a69092c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 16:36:48.254838 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:48.254796 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:36:48.255092 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:48.254863 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" podUID="8c8da48a-94af-4d09-aef7-21c1a69092c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 16:36:49.255764 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:49.255692 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:36:49.256194 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:49.255786 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" podUID="8c8da48a-94af-4d09-aef7-21c1a69092c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 16:36:50.255217 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:50.255178 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Nov 03 16:36:50.255424 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:50.255240 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" podUID="8c8da48a-94af-4d09-aef7-21c1a69092c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Nov 03 16:36:51.496543 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:51.496506 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:51.497612 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:51.497491 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:36:51.498992 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:36:51.498973 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:37:00.131604 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:37:00.131558 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:37:20.662660 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:20.662623 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:37:26.502985 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.502946 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr"] Nov 03 16:37:26.508741 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.508716 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.527730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.527689 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr"] Nov 03 16:37:26.539068 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.539043 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7519b7\"/\"openshift-service-ca.crt\"" Nov 03 16:37:26.539068 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.539046 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7519b7\"/\"default-dockercfg-v9hmw\"" Nov 03 16:37:26.539233 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.539125 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7519b7\"/\"kube-root-ca.crt\"" Nov 03 16:37:26.539233 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.539145 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7519b7\"/\"e2e-conflict-a-f7f1d684-kserve-self-signed-certs\"" Nov 03 16:37:26.613199 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.613162 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-home\") pod \"e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.613367 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.613277 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-model-cache\") pod \"e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.613367 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.613313 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxhvp\" (UniqueName: \"kubernetes.io/projected/f0499181-e534-4622-9c4c-96c834042bb8-kube-api-access-lxhvp\") pod \"e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.613509 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.613490 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0499181-e534-4622-9c4c-96c834042bb8-tls-certs\") pod \"e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.613569 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.613538 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-dshm\") pod \"e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.613608 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.613594 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-tmp-dir\") pod \"e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.734721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.717122 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-model-cache\") pod \"e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.734721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.717177 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhvp\" (UniqueName: \"kubernetes.io/projected/f0499181-e534-4622-9c4c-96c834042bb8-kube-api-access-lxhvp\") pod \"e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.734721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.717289 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0499181-e534-4622-9c4c-96c834042bb8-tls-certs\") pod \"e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.734721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.717325 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-dshm\") pod \"e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.734721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.717353 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-tmp-dir\") pod \"e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.734721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.717422 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-home\") pod \"e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.734721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.717945 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-home\") pod \"e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.734721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.718181 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-model-cache\") pod \"e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.734721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.719422 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-tmp-dir\") pod \"e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.743157 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.743122 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0499181-e534-4622-9c4c-96c834042bb8-tls-certs\") pod \"e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.744448 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.744408 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-dshm\") pod \"e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.783776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.778716 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxhvp\" (UniqueName: \"kubernetes.io/projected/f0499181-e534-4622-9c4c-96c834042bb8-kube-api-access-lxhvp\") pod \"e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:26.821762 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:26.821724 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:27.135868 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:27.135838 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr"] Nov 03 16:37:27.136354 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:37:27.136329 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0499181_e534_4622_9c4c_96c834042bb8.slice/crio-2ba95b57f3a15ddf27adb72a2a8e216a076d890ccb014286ff5c620a07ccb9b6 WatchSource:0}: Error finding container 2ba95b57f3a15ddf27adb72a2a8e216a076d890ccb014286ff5c620a07ccb9b6: Status 404 returned error can't find the container with id 2ba95b57f3a15ddf27adb72a2a8e216a076d890ccb014286ff5c620a07ccb9b6 Nov 03 16:37:27.408064 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:27.407953 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" event={"ID":"f0499181-e534-4622-9c4c-96c834042bb8","Type":"ContainerStarted","Data":"54a4d418112cb9e7e314d43fa8ec146fe5febc8bc687800152d08124babd2002"} Nov 03 16:37:27.408064 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:27.408004 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" event={"ID":"f0499181-e534-4622-9c4c-96c834042bb8","Type":"ContainerStarted","Data":"2ba95b57f3a15ddf27adb72a2a8e216a076d890ccb014286ff5c620a07ccb9b6"} Nov 03 16:37:27.454972 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:27.454926 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" podStartSLOduration=1.45491251 podStartE2EDuration="1.45491251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:27.453483513 +0000 UTC m=+1391.042483485" watchObservedRunningTime="2026-11-03 16:37:27.45491251 +0000 UTC m=+1391.043912479" Nov 03 16:37:27.822551 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:27.822512 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:30.193437 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:37:30.193395 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:37:35.784972 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.784939 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6"] Nov 03 16:37:35.788938 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.788914 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.807518 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.807481 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-71d7b4-openshift-default-dockercfg-5vhxc\"" Nov 03 16:37:35.824458 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.824424 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6"] Nov 03 16:37:35.838465 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.838438 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.838641 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.838499 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.838687 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.838661 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.838851 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.838830 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.838903 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.838866 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.838903 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.838897 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.839083 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.839002 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpldk\" (UniqueName: \"kubernetes.io/projected/c802c606-eb7c-497f-a219-a4fc5ef34cd1-kube-api-access-kpldk\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.839083 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.839045 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.839401 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.839309 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.940439 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.940406 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.940439 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.940445 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.940656 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.940473 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.940656 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.940507 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.940656 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.940578 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.940656 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.940600 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.940656 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.940627 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.940984 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.940671 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpldk\" (UniqueName: \"kubernetes.io/projected/c802c606-eb7c-497f-a219-a4fc5ef34cd1-kube-api-access-kpldk\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.940984 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.940723 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.941093 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.940999 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.941197 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.941163 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.941197 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.941198 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.941550 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.941372 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.941550 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.941443 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.943062 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.943043 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.943382 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.943359 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.965276 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.965243 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:35.965408 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:35.965360 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpldk\" (UniqueName: \"kubernetes.io/projected/c802c606-eb7c-497f-a219-a4fc5ef34cd1-kube-api-access-kpldk\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:36.102747 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:36.102637 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:36.425823 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:36.425795 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6"] Nov 03 16:37:36.431030 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:37:36.430990 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc802c606_eb7c_497f_a219_a4fc5ef34cd1.slice/crio-7fb3c92304bc1c5239154f408095f6fed89fe7b95513a8af2781194ccb48e032 WatchSource:0}: Error finding container 7fb3c92304bc1c5239154f408095f6fed89fe7b95513a8af2781194ccb48e032: Status 404 returned error can't find the container with id 7fb3c92304bc1c5239154f408095f6fed89fe7b95513a8af2781194ccb48e032 Nov 03 16:37:36.432766 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:36.432733 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:37:36.432840 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:36.432802 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:37:36.432878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:36.432848 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:37:36.480005 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:36.479972 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" event={"ID":"c802c606-eb7c-497f-a219-a4fc5ef34cd1","Type":"ContainerStarted","Data":"7fb3c92304bc1c5239154f408095f6fed89fe7b95513a8af2781194ccb48e032"} Nov 03 16:37:36.822121 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:36.822074 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:36.838475 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:36.838442 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:37.487357 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:37.487322 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" event={"ID":"c802c606-eb7c-497f-a219-a4fc5ef34cd1","Type":"ContainerStarted","Data":"5535f69d708bf7a355ab210abcd2b1bf3e86f020e623dfb569827f81c1290272"} Nov 03 16:37:37.500177 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:37.500148 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:37:37.549284 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:37.549222 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" podStartSLOduration=2.5492089079999998 podStartE2EDuration="2.549208908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:37.547086752 +0000 UTC m=+1401.136086722" watchObservedRunningTime="2026-11-03 16:37:37.549208908 +0000 UTC m=+1401.138208959" Nov 03 16:37:38.118724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:38.113440 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:38.122841 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:38.121611 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 16:37:38.122841 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:38.121682 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" podUID="c802c606-eb7c-497f-a219-a4fc5ef34cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 16:37:39.017722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:39.012720 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:37:39.117727 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:39.107105 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 16:37:39.117727 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:39.107183 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" podUID="c802c606-eb7c-497f-a219-a4fc5ef34cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 16:37:40.104725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:40.104112 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 16:37:40.104725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:40.104173 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" podUID="c802c606-eb7c-497f-a219-a4fc5ef34cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 16:37:41.124750 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:41.123888 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 16:37:41.124750 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:41.123971 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" podUID="c802c606-eb7c-497f-a219-a4fc5ef34cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 16:37:42.107308 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:42.107268 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 16:37:42.107627 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:42.107589 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" podUID="c802c606-eb7c-497f-a219-a4fc5ef34cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 16:37:43.026726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:43.025818 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 03 16:37:43.103548 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:43.103503 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Nov 03 16:37:43.103912 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:43.103879 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" podUID="c802c606-eb7c-497f-a219-a4fc5ef34cd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Nov 03 16:37:44.108381 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:44.108352 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:44.108827 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:44.108641 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:44.109351 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:44.109332 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:37:49.630813 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.630782 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6"] Nov 03 16:37:49.635910 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.635893 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" Nov 03 16:37:49.657997 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.657971 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 16:37:49.660985 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.660957 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6"] Nov 03 16:37:49.668604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.668582 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-qdtlb\"" Nov 03 16:37:49.668604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.668582 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-061ea8950c76\"" Nov 03 16:37:49.671416 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.671393 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 16:37:49.671530 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.671471 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 16:37:49.671530 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.671476 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 16:37:49.684887 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.684862 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f476a-5b35-4073-9a49-b060db425ac8-plugins-config-volume\") pod \"payload-processing-external-model-c7d4884db-gvlp6\" (UID: \"5f3f476a-5b35-4073-9a49-b060db425ac8\") " pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" Nov 03 16:37:49.684968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.684905 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr78d\" (UniqueName: \"kubernetes.io/projected/5f3f476a-5b35-4073-9a49-b060db425ac8-kube-api-access-fr78d\") pod \"payload-processing-external-model-c7d4884db-gvlp6\" (UID: \"5f3f476a-5b35-4073-9a49-b060db425ac8\") " pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" Nov 03 16:37:49.685106 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.685087 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f476a-5b35-4073-9a49-b060db425ac8-routing-overlay-volume\") pod \"payload-processing-external-model-c7d4884db-gvlp6\" (UID: \"5f3f476a-5b35-4073-9a49-b060db425ac8\") " pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" Nov 03 16:37:49.685154 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.685126 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f3f476a-5b35-4073-9a49-b060db425ac8-provider-credentials\") pod \"payload-processing-external-model-c7d4884db-gvlp6\" (UID: \"5f3f476a-5b35-4073-9a49-b060db425ac8\") " pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" Nov 03 16:37:49.785766 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.785736 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f476a-5b35-4073-9a49-b060db425ac8-plugins-config-volume\") pod \"payload-processing-external-model-c7d4884db-gvlp6\" (UID: \"5f3f476a-5b35-4073-9a49-b060db425ac8\") " pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" Nov 03 16:37:49.785928 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.785777 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr78d\" (UniqueName: \"kubernetes.io/projected/5f3f476a-5b35-4073-9a49-b060db425ac8-kube-api-access-fr78d\") pod \"payload-processing-external-model-c7d4884db-gvlp6\" (UID: \"5f3f476a-5b35-4073-9a49-b060db425ac8\") " pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" Nov 03 16:37:49.785928 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.785864 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f476a-5b35-4073-9a49-b060db425ac8-routing-overlay-volume\") pod \"payload-processing-external-model-c7d4884db-gvlp6\" (UID: \"5f3f476a-5b35-4073-9a49-b060db425ac8\") " pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" Nov 03 16:37:49.785928 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.785886 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f3f476a-5b35-4073-9a49-b060db425ac8-provider-credentials\") pod \"payload-processing-external-model-c7d4884db-gvlp6\" (UID: \"5f3f476a-5b35-4073-9a49-b060db425ac8\") " pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" Nov 03 16:37:49.786459 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.786433 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f476a-5b35-4073-9a49-b060db425ac8-routing-overlay-volume\") pod \"payload-processing-external-model-c7d4884db-gvlp6\" (UID: \"5f3f476a-5b35-4073-9a49-b060db425ac8\") " pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" Nov 03 16:37:49.786543 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.786514 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f476a-5b35-4073-9a49-b060db425ac8-plugins-config-volume\") pod \"payload-processing-external-model-c7d4884db-gvlp6\" (UID: \"5f3f476a-5b35-4073-9a49-b060db425ac8\") " pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" Nov 03 16:37:49.788428 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.788406 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f3f476a-5b35-4073-9a49-b060db425ac8-provider-credentials\") pod \"payload-processing-external-model-c7d4884db-gvlp6\" (UID: \"5f3f476a-5b35-4073-9a49-b060db425ac8\") " pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" Nov 03 16:37:49.811183 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.811155 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr78d\" (UniqueName: \"kubernetes.io/projected/5f3f476a-5b35-4073-9a49-b060db425ac8-kube-api-access-fr78d\") pod \"payload-processing-external-model-c7d4884db-gvlp6\" (UID: \"5f3f476a-5b35-4073-9a49-b060db425ac8\") " pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" Nov 03 16:37:49.950425 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:49.950332 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" Nov 03 16:37:50.265898 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:50.265868 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6"] Nov 03 16:37:50.266910 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:37:50.266884 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f3f476a_5b35_4073_9a49_b060db425ac8.slice/crio-cff1badb6b39ae54647d200678f477c895a8298ccaf15f825c141f2102eea68e WatchSource:0}: Error finding container cff1badb6b39ae54647d200678f477c895a8298ccaf15f825c141f2102eea68e: Status 404 returned error can't find the container with id cff1badb6b39ae54647d200678f477c895a8298ccaf15f825c141f2102eea68e Nov 03 16:37:50.647908 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:50.647824 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" event={"ID":"5f3f476a-5b35-4073-9a49-b060db425ac8","Type":"ContainerStarted","Data":"91376025fe9b166c981e7c451aebae2fe67a1fe975a11891bf5a8dcc9c4d2b3a"} Nov 03 16:37:50.647908 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:50.647885 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" event={"ID":"5f3f476a-5b35-4073-9a49-b060db425ac8","Type":"ContainerStarted","Data":"cff1badb6b39ae54647d200678f477c895a8298ccaf15f825c141f2102eea68e"} Nov 03 16:37:50.648333 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:50.648054 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" Nov 03 16:37:50.694496 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:50.694446 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" podStartSLOduration=1.694433889 podStartE2EDuration="1.694433889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:37:50.69253327 +0000 UTC m=+1414.281533238" watchObservedRunningTime="2026-11-03 16:37:50.694433889 +0000 UTC m=+1414.283433857" Nov 03 16:37:54.197916 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:54.197881 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:37:56.151908 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:56.151866 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6"] Nov 03 16:37:56.156981 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:37:56.156951 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" Nov 03 16:38:00.129921 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:38:00.129881 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:38:01.079551 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.079503 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk"] Nov 03 16:38:01.084873 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.084851 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.130203 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.130174 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk"] Nov 03 16:38:01.153116 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.153080 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" podUID="5f3f476a-5b35-4073-9a49-b060db425ac8" containerName="payload-processing" containerID="cri-o://91376025fe9b166c981e7c451aebae2fe67a1fe975a11891bf5a8dcc9c4d2b3a" gracePeriod=25 Nov 03 16:38:01.170690 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.170664 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.170818 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.170799 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.170871 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.170831 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.170871 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.170850 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pttrw\" (UniqueName: \"kubernetes.io/projected/ee442df4-03ea-4f56-9b1a-a31e84d2c861-kube-api-access-pttrw\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.170960 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.170887 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.171005 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.170984 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.171054 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.171041 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.171089 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.171063 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.171128 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.171102 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.272202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.272171 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.272339 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.272233 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.272339 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.272252 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.272339 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.272278 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pttrw\" (UniqueName: \"kubernetes.io/projected/ee442df4-03ea-4f56-9b1a-a31e84d2c861-kube-api-access-pttrw\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.272339 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.272308 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.272339 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.272332 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.272620 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.272434 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.272620 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.272457 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.272620 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.272524 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.272843 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.272623 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.272999 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.272978 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.272999 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.272990 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.273198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.273175 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.273274 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.273203 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.275124 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.275096 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.275252 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.275232 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.296893 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.296856 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.297271 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.297248 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pttrw\" (UniqueName: \"kubernetes.io/projected/ee442df4-03ea-4f56-9b1a-a31e84d2c861-kube-api-access-pttrw\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.398909 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.398880 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:01.405124 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.405099 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" Nov 03 16:38:01.475122 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.475089 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f3f476a-5b35-4073-9a49-b060db425ac8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f3f476a-5b35-4073-9a49-b060db425ac8-provider-credentials\") pod \"5f3f476a-5b35-4073-9a49-b060db425ac8\" (UID: \"5f3f476a-5b35-4073-9a49-b060db425ac8\") " Nov 03 16:38:01.475270 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.475212 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f3f476a-5b35-4073-9a49-b060db425ac8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f476a-5b35-4073-9a49-b060db425ac8-routing-overlay-volume\") pod \"5f3f476a-5b35-4073-9a49-b060db425ac8\" (UID: \"5f3f476a-5b35-4073-9a49-b060db425ac8\") " Nov 03 16:38:01.475316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.475277 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f3f476a-5b35-4073-9a49-b060db425ac8-kube-api-access-fr78d\" (UniqueName: \"kubernetes.io/projected/5f3f476a-5b35-4073-9a49-b060db425ac8-kube-api-access-fr78d\") pod \"5f3f476a-5b35-4073-9a49-b060db425ac8\" (UID: \"5f3f476a-5b35-4073-9a49-b060db425ac8\") " Nov 03 16:38:01.475350 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.475331 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f3f476a-5b35-4073-9a49-b060db425ac8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f476a-5b35-4073-9a49-b060db425ac8-plugins-config-volume\") pod \"5f3f476a-5b35-4073-9a49-b060db425ac8\" (UID: \"5f3f476a-5b35-4073-9a49-b060db425ac8\") " Nov 03 16:38:01.475921 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.475885 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3f476a-5b35-4073-9a49-b060db425ac8-routing-overlay-volume" pod "5f3f476a-5b35-4073-9a49-b060db425ac8" (UID: "5f3f476a-5b35-4073-9a49-b060db425ac8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:38:01.476201 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.476177 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f3f476a-5b35-4073-9a49-b060db425ac8-plugins-config-volume" pod "5f3f476a-5b35-4073-9a49-b060db425ac8" (UID: "5f3f476a-5b35-4073-9a49-b060db425ac8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:38:01.478493 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.478467 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3f476a-5b35-4073-9a49-b060db425ac8-provider-credentials" pod "5f3f476a-5b35-4073-9a49-b060db425ac8" (UID: "5f3f476a-5b35-4073-9a49-b060db425ac8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:01.479505 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.479485 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3f476a-5b35-4073-9a49-b060db425ac8-kube-api-access-fr78d" pod "5f3f476a-5b35-4073-9a49-b060db425ac8" (UID: "5f3f476a-5b35-4073-9a49-b060db425ac8"). InnerVolumeSpecName "kube-api-access-fr78d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:01.576788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.576756 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f3f476a-5b35-4073-9a49-b060db425ac8-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:38:01.576788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.576784 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f476a-5b35-4073-9a49-b060db425ac8-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:38:01.576788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.576793 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr78d\" (UniqueName: \"kubernetes.io/projected/5f3f476a-5b35-4073-9a49-b060db425ac8-kube-api-access-fr78d\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:38:01.577022 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.576805 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f3f476a-5b35-4073-9a49-b060db425ac8-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:38:01.726669 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.726641 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk"] Nov 03 16:38:01.727346 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:38:01.727316 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee442df4_03ea_4f56_9b1a_a31e84d2c861.slice/crio-e192172f142ca83b6efbd489ef6c1ab591ee373f6e40eb5a3b30dd2c6b0a7caa WatchSource:0}: Error finding container e192172f142ca83b6efbd489ef6c1ab591ee373f6e40eb5a3b30dd2c6b0a7caa: Status 404 returned error can't find the container with id e192172f142ca83b6efbd489ef6c1ab591ee373f6e40eb5a3b30dd2c6b0a7caa Nov 03 16:38:01.729952 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.729911 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:38:01.730053 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.730002 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:38:01.730053 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.730041 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:38:01.786014 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.785987 4947 generic.go:356] "Generic (PLEG): container finished" podID="5f3f476a-5b35-4073-9a49-b060db425ac8" containerID="91376025fe9b166c981e7c451aebae2fe67a1fe975a11891bf5a8dcc9c4d2b3a" exitCode=0 Nov 03 16:38:01.786123 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.786108 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" Nov 03 16:38:01.786178 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.786125 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" event={"ID":"5f3f476a-5b35-4073-9a49-b060db425ac8","Type":"ContainerDied","Data":"91376025fe9b166c981e7c451aebae2fe67a1fe975a11891bf5a8dcc9c4d2b3a"} Nov 03 16:38:01.786235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.786183 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6" event={"ID":"5f3f476a-5b35-4073-9a49-b060db425ac8","Type":"ContainerDied","Data":"cff1badb6b39ae54647d200678f477c895a8298ccaf15f825c141f2102eea68e"} Nov 03 16:38:01.786235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.786211 4947 scope.go:122] "RemoveContainer" containerID="91376025fe9b166c981e7c451aebae2fe67a1fe975a11891bf5a8dcc9c4d2b3a" Nov 03 16:38:01.787922 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.787899 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" event={"ID":"ee442df4-03ea-4f56-9b1a-a31e84d2c861","Type":"ContainerStarted","Data":"e192172f142ca83b6efbd489ef6c1ab591ee373f6e40eb5a3b30dd2c6b0a7caa"} Nov 03 16:38:01.794899 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.794880 4947 scope.go:122] "RemoveContainer" containerID="91376025fe9b166c981e7c451aebae2fe67a1fe975a11891bf5a8dcc9c4d2b3a" Nov 03 16:38:01.795147 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:38:01.795124 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91376025fe9b166c981e7c451aebae2fe67a1fe975a11891bf5a8dcc9c4d2b3a\": container with ID starting with 91376025fe9b166c981e7c451aebae2fe67a1fe975a11891bf5a8dcc9c4d2b3a not found: ID does not exist" containerID="91376025fe9b166c981e7c451aebae2fe67a1fe975a11891bf5a8dcc9c4d2b3a" Nov 03 16:38:01.795220 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.795161 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91376025fe9b166c981e7c451aebae2fe67a1fe975a11891bf5a8dcc9c4d2b3a"} err="failed to get container status \"91376025fe9b166c981e7c451aebae2fe67a1fe975a11891bf5a8dcc9c4d2b3a\": rpc error: code = NotFound desc = could not find container \"91376025fe9b166c981e7c451aebae2fe67a1fe975a11891bf5a8dcc9c4d2b3a\": container with ID starting with 91376025fe9b166c981e7c451aebae2fe67a1fe975a11891bf5a8dcc9c4d2b3a not found: ID does not exist" Nov 03 16:38:01.842666 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.842631 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6"] Nov 03 16:38:01.845799 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:01.845773 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-c7d4884db-gvlp6"] Nov 03 16:38:02.564562 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:02.564531 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f3f476a-5b35-4073-9a49-b060db425ac8" path="/var/lib/kubelet/pods/5f3f476a-5b35-4073-9a49-b060db425ac8/volumes" Nov 03 16:38:02.797378 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:02.797343 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" event={"ID":"ee442df4-03ea-4f56-9b1a-a31e84d2c861","Type":"ContainerStarted","Data":"02d52b1bdea1129c16dbce0bf1bec3054d85f67bf104fa9e1389a6232ef272d4"} Nov 03 16:38:02.862081 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:02.861963 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" podStartSLOduration=1.861944849 podStartE2EDuration="1.861944849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:02.859441389 +0000 UTC m=+1426.448441360" watchObservedRunningTime="2026-11-03 16:38:02.861944849 +0000 UTC m=+1426.450944818" Nov 03 16:38:03.421725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:03.421397 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:03.424267 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:03.424237 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:38:03.424389 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:03.424297 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" podUID="ee442df4-03ea-4f56-9b1a-a31e84d2c861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 16:38:04.417829 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:04.417793 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:38:04.418436 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:04.418402 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" podUID="ee442df4-03ea-4f56-9b1a-a31e84d2c861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 16:38:05.403158 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:05.403120 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:38:05.403422 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:05.403394 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" podUID="ee442df4-03ea-4f56-9b1a-a31e84d2c861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 16:38:06.422858 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:06.422816 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:38:06.423419 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:06.422884 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" podUID="ee442df4-03ea-4f56-9b1a-a31e84d2c861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 16:38:07.401729 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:07.400025 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:38:07.401729 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:07.400085 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" podUID="ee442df4-03ea-4f56-9b1a-a31e84d2c861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 16:38:08.399642 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.399603 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body= Nov 03 16:38:08.400151 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.399665 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" podUID="ee442df4-03ea-4f56-9b1a-a31e84d2c861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" Nov 03 16:38:08.556933 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.553677 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r"] Nov 03 16:38:08.556933 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.554681 4947 cpu_manager.go:414] "containerMap: removing container" podUID="5f3f476a-5b35-4073-9a49-b060db425ac8" containerName="payload-processing" Nov 03 16:38:08.556933 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.554735 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3f476a-5b35-4073-9a49-b060db425ac8" containerName="payload-processing" Nov 03 16:38:08.556933 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.555044 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f3f476a-5b35-4073-9a49-b060db425ac8" containerName="payload-processing" Nov 03 16:38:08.561333 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.559441 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.620728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.591847 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-460104\"/\"kube-root-ca.crt\"" Nov 03 16:38:08.620728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.591958 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-460104\"/\"distinct-w4-460104-kserve-self-signed-certs\"" Nov 03 16:38:08.635105 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.634035 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-460104\"/\"default-dockercfg-2xrq7\"" Nov 03 16:38:08.635105 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.634138 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-460104\"/\"openshift-service-ca.crt\"" Nov 03 16:38:08.696364 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.696274 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-model-cache\") pod \"distinct-w4-460104-kserve-84bd9f98d4-rcl5r\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.696592 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.696572 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40eb248-b0bd-43ab-821c-b21ad6f8a364-tls-certs\") pod \"distinct-w4-460104-kserve-84bd9f98d4-rcl5r\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.696844 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.696823 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-tmp-dir\") pod \"distinct-w4-460104-kserve-84bd9f98d4-rcl5r\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.697016 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.696998 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-dshm\") pod \"distinct-w4-460104-kserve-84bd9f98d4-rcl5r\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.697208 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.697191 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qxkz\" (UniqueName: \"kubernetes.io/projected/f40eb248-b0bd-43ab-821c-b21ad6f8a364-kube-api-access-2qxkz\") pod \"distinct-w4-460104-kserve-84bd9f98d4-rcl5r\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.697392 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.697375 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-home\") pod \"distinct-w4-460104-kserve-84bd9f98d4-rcl5r\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.800653 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.800608 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-home\") pod \"distinct-w4-460104-kserve-84bd9f98d4-rcl5r\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.824903 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.822187 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-home\") pod \"distinct-w4-460104-kserve-84bd9f98d4-rcl5r\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.825317 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.825275 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-model-cache\") pod \"distinct-w4-460104-kserve-84bd9f98d4-rcl5r\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.826085 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.826055 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40eb248-b0bd-43ab-821c-b21ad6f8a364-tls-certs\") pod \"distinct-w4-460104-kserve-84bd9f98d4-rcl5r\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.837950 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.837916 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-tmp-dir\") pod \"distinct-w4-460104-kserve-84bd9f98d4-rcl5r\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.838598 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.838575 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-dshm\") pod \"distinct-w4-460104-kserve-84bd9f98d4-rcl5r\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.839232 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.839210 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qxkz\" (UniqueName: \"kubernetes.io/projected/f40eb248-b0bd-43ab-821c-b21ad6f8a364-kube-api-access-2qxkz\") pod \"distinct-w4-460104-kserve-84bd9f98d4-rcl5r\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.896909 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.825982 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-model-cache\") pod \"distinct-w4-460104-kserve-84bd9f98d4-rcl5r\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.897147 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.838479 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-tmp-dir\") pod \"distinct-w4-460104-kserve-84bd9f98d4-rcl5r\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.897259 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.850064 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-dshm\") pod \"distinct-w4-460104-kserve-84bd9f98d4-rcl5r\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.897351 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.880360 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40eb248-b0bd-43ab-821c-b21ad6f8a364-tls-certs\") pod \"distinct-w4-460104-kserve-84bd9f98d4-rcl5r\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.902347 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.902320 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r"] Nov 03 16:38:08.905894 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.905832 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qxkz\" (UniqueName: \"kubernetes.io/projected/f40eb248-b0bd-43ab-821c-b21ad6f8a364-kube-api-access-2qxkz\") pod \"distinct-w4-460104-kserve-84bd9f98d4-rcl5r\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:08.955374 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:08.955025 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:09.489238 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:09.489159 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r"] Nov 03 16:38:09.924556 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:09.924472 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" event={"ID":"f40eb248-b0bd-43ab-821c-b21ad6f8a364","Type":"ContainerStarted","Data":"b415798784cf61b5a9982f88d13b3d1b4599b748ae1ceb74dfa299f1fd000c6a"} Nov 03 16:38:10.481723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:10.467796 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:38:10.481723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:10.467873 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" podUID="ee442df4-03ea-4f56-9b1a-a31e84d2c861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:38:10.965722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:10.965665 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" event={"ID":"f40eb248-b0bd-43ab-821c-b21ad6f8a364","Type":"ContainerStarted","Data":"2fb53936824fca8eaa6ecd6f840d476533c01fdccedcb54acdcbb3063f83fde0"} Nov 03 16:38:11.215775 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:11.209827 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:11.215775 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:11.211254 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:11.218388 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:11.217952 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 16:38:11.263605 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:11.263542 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podStartSLOduration=3.263523148 podStartE2EDuration="3.263523148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:11.020920368 +0000 UTC m=+1434.609920338" watchObservedRunningTime="2026-11-03 16:38:11.263523148 +0000 UTC m=+1434.852523121" Nov 03 16:38:11.959191 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:11.958084 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:16.017619 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:16.017582 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:38:18.956053 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:18.956015 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:18.968979 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:18.968949 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:19.020872 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:19.020838 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:38:26.268932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:26.268897 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr"] Nov 03 16:38:30.144158 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:38:30.144112 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:38:35.655815 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:35.655777 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:38:41.290004 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.289966 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" podUID="f0499181-e534-4622-9c4c-96c834042bb8" containerName="main" containerID="cri-o://54a4d418112cb9e7e314d43fa8ec146fe5febc8bc687800152d08124babd2002" gracePeriod=45 Nov 03 16:38:41.564573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.564550 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:38:41.629678 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.629642 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0499181-e534-4622-9c4c-96c834042bb8-kube-api-access-lxhvp\" (UniqueName: \"kubernetes.io/projected/f0499181-e534-4622-9c4c-96c834042bb8-kube-api-access-lxhvp\") pod \"f0499181-e534-4622-9c4c-96c834042bb8\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " Nov 03 16:38:41.629678 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.629678 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-model-cache\") pod \"f0499181-e534-4622-9c4c-96c834042bb8\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " Nov 03 16:38:41.630034 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.629721 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-tmp-dir\") pod \"f0499181-e534-4622-9c4c-96c834042bb8\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " Nov 03 16:38:41.630034 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.629757 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0499181-e534-4622-9c4c-96c834042bb8-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0499181-e534-4622-9c4c-96c834042bb8-tls-certs\") pod \"f0499181-e534-4622-9c4c-96c834042bb8\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " Nov 03 16:38:41.630034 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.629880 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-dshm\") pod \"f0499181-e534-4622-9c4c-96c834042bb8\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " Nov 03 16:38:41.630034 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.629941 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-home\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-home\") pod \"f0499181-e534-4622-9c4c-96c834042bb8\" (UID: \"f0499181-e534-4622-9c4c-96c834042bb8\") " Nov 03 16:38:41.630034 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.629989 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-tmp-dir" pod "f0499181-e534-4622-9c4c-96c834042bb8" (UID: "f0499181-e534-4622-9c4c-96c834042bb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:38:41.630305 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.630280 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-model-cache" pod "f0499181-e534-4622-9c4c-96c834042bb8" (UID: "f0499181-e534-4622-9c4c-96c834042bb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:38:41.630368 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.630317 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-home" pod "f0499181-e534-4622-9c4c-96c834042bb8" (UID: "f0499181-e534-4622-9c4c-96c834042bb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:38:41.630535 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.630514 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:38:41.630840 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.630538 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:38:41.630840 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.630551 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:38:41.632429 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.632406 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0499181-e534-4622-9c4c-96c834042bb8-kube-api-access-lxhvp" pod "f0499181-e534-4622-9c4c-96c834042bb8" (UID: "f0499181-e534-4622-9c4c-96c834042bb8"). InnerVolumeSpecName "kube-api-access-lxhvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:38:41.632645 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.632618 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-dshm" pod "f0499181-e534-4622-9c4c-96c834042bb8" (UID: "f0499181-e534-4622-9c4c-96c834042bb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:38:41.632784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.632726 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0499181-e534-4622-9c4c-96c834042bb8-tls-certs" pod "f0499181-e534-4622-9c4c-96c834042bb8" (UID: "f0499181-e534-4622-9c4c-96c834042bb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:38:41.735914 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.735765 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxhvp\" (UniqueName: \"kubernetes.io/projected/f0499181-e534-4622-9c4c-96c834042bb8-kube-api-access-lxhvp\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:38:41.735914 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.735799 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0499181-e534-4622-9c4c-96c834042bb8-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:38:41.735914 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:41.735814 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0499181-e534-4622-9c4c-96c834042bb8-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:38:42.215960 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:42.215921 4947 generic.go:356] "Generic (PLEG): container finished" podID="f0499181-e534-4622-9c4c-96c834042bb8" containerID="54a4d418112cb9e7e314d43fa8ec146fe5febc8bc687800152d08124babd2002" exitCode=0 Nov 03 16:38:42.215960 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:42.215966 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" Nov 03 16:38:42.216170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:42.215992 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" event={"ID":"f0499181-e534-4622-9c4c-96c834042bb8","Type":"ContainerDied","Data":"54a4d418112cb9e7e314d43fa8ec146fe5febc8bc687800152d08124babd2002"} Nov 03 16:38:42.216170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:42.216029 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" event={"ID":"f0499181-e534-4622-9c4c-96c834042bb8","Type":"ContainerDied","Data":"2ba95b57f3a15ddf27adb72a2a8e216a076d890ccb014286ff5c620a07ccb9b6"} Nov 03 16:38:42.216170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:42.216051 4947 scope.go:122] "RemoveContainer" containerID="54a4d418112cb9e7e314d43fa8ec146fe5febc8bc687800152d08124babd2002" Nov 03 16:38:42.226021 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:42.226000 4947 scope.go:122] "RemoveContainer" containerID="54a4d418112cb9e7e314d43fa8ec146fe5febc8bc687800152d08124babd2002" Nov 03 16:38:42.226290 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:38:42.226270 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54a4d418112cb9e7e314d43fa8ec146fe5febc8bc687800152d08124babd2002\": container with ID starting with 54a4d418112cb9e7e314d43fa8ec146fe5febc8bc687800152d08124babd2002 not found: ID does not exist" containerID="54a4d418112cb9e7e314d43fa8ec146fe5febc8bc687800152d08124babd2002" Nov 03 16:38:42.226368 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:42.226299 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a4d418112cb9e7e314d43fa8ec146fe5febc8bc687800152d08124babd2002"} err="failed to get container status \"54a4d418112cb9e7e314d43fa8ec146fe5febc8bc687800152d08124babd2002\": rpc error: code = NotFound desc = could not find container \"54a4d418112cb9e7e314d43fa8ec146fe5febc8bc687800152d08124babd2002\": container with ID starting with 54a4d418112cb9e7e314d43fa8ec146fe5febc8bc687800152d08124babd2002 not found: ID does not exist" Nov 03 16:38:42.271826 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:42.271799 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr"] Nov 03 16:38:42.291015 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:42.290991 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr"] Nov 03 16:38:42.488615 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:42.488518 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/e2e-conflict-a-f7f1d684-kserve-f8fb4f9f-vv9mr" podUID="f0499181-e534-4622-9c4c-96c834042bb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 16:38:42.565095 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:42.565058 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0499181-e534-4622-9c4c-96c834042bb8" path="/var/lib/kubelet/pods/f0499181-e534-4622-9c4c-96c834042bb8/volumes" Nov 03 16:38:43.477880 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.477846 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68"] Nov 03 16:38:43.478634 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.478614 4947 cpu_manager.go:414] "containerMap: removing container" podUID="f0499181-e534-4622-9c4c-96c834042bb8" containerName="main" Nov 03 16:38:43.478679 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.478638 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0499181-e534-4622-9c4c-96c834042bb8" containerName="main" Nov 03 16:38:43.478861 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.478847 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0499181-e534-4622-9c4c-96c834042bb8" containerName="main" Nov 03 16:38:43.483846 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.483827 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.500748 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.500719 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68"] Nov 03 16:38:43.512010 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.511985 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7519b7\"/\"distinct-w2-7519b7-kserve-self-signed-certs\"" Nov 03 16:38:43.512146 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.512094 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7519b7\"/\"default-dockercfg-v9hmw\"" Nov 03 16:38:43.554406 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.554374 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-tmp-dir\") pod \"distinct-w2-7519b7-kserve-59f85cc7f8-kfl68\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.554406 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.554407 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlnqv\" (UniqueName: \"kubernetes.io/projected/3a46652a-73e1-4a0e-95ab-668bb2126992-kube-api-access-mlnqv\") pod \"distinct-w2-7519b7-kserve-59f85cc7f8-kfl68\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.554586 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.554443 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a46652a-73e1-4a0e-95ab-668bb2126992-tls-certs\") pod \"distinct-w2-7519b7-kserve-59f85cc7f8-kfl68\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.554586 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.554530 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-dshm\") pod \"distinct-w2-7519b7-kserve-59f85cc7f8-kfl68\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.554586 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.554555 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-model-cache\") pod \"distinct-w2-7519b7-kserve-59f85cc7f8-kfl68\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.554586 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.554583 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-home\") pod \"distinct-w2-7519b7-kserve-59f85cc7f8-kfl68\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.595125 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.595092 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7519b7\"/\"openshift-service-ca.crt\"" Nov 03 16:38:43.595125 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.595123 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7519b7\"/\"kube-root-ca.crt\"" Nov 03 16:38:43.663369 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.661377 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-tmp-dir\") pod \"distinct-w2-7519b7-kserve-59f85cc7f8-kfl68\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.663369 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.661447 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlnqv\" (UniqueName: \"kubernetes.io/projected/3a46652a-73e1-4a0e-95ab-668bb2126992-kube-api-access-mlnqv\") pod \"distinct-w2-7519b7-kserve-59f85cc7f8-kfl68\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.663369 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.661536 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a46652a-73e1-4a0e-95ab-668bb2126992-tls-certs\") pod \"distinct-w2-7519b7-kserve-59f85cc7f8-kfl68\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.663369 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.661600 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-dshm\") pod \"distinct-w2-7519b7-kserve-59f85cc7f8-kfl68\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.663369 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.661649 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-model-cache\") pod \"distinct-w2-7519b7-kserve-59f85cc7f8-kfl68\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.663369 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.661729 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-home\") pod \"distinct-w2-7519b7-kserve-59f85cc7f8-kfl68\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.663369 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.662342 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-home\") pod \"distinct-w2-7519b7-kserve-59f85cc7f8-kfl68\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.663369 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.662601 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-tmp-dir\") pod \"distinct-w2-7519b7-kserve-59f85cc7f8-kfl68\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.668572 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.668542 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-model-cache\") pod \"distinct-w2-7519b7-kserve-59f85cc7f8-kfl68\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.689719 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.688250 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a46652a-73e1-4a0e-95ab-668bb2126992-tls-certs\") pod \"distinct-w2-7519b7-kserve-59f85cc7f8-kfl68\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.695769 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.695737 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-dshm\") pod \"distinct-w2-7519b7-kserve-59f85cc7f8-kfl68\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.696679 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.696620 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlnqv\" (UniqueName: \"kubernetes.io/projected/3a46652a-73e1-4a0e-95ab-668bb2126992-kube-api-access-mlnqv\") pod \"distinct-w2-7519b7-kserve-59f85cc7f8-kfl68\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:43.797237 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:43.797206 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:44.102487 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:44.102458 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68"] Nov 03 16:38:44.102885 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:38:44.102865 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a46652a_73e1_4a0e_95ab_668bb2126992.slice/crio-6e7f4a8e2946f01d25622df7a97916ad52ae6f363aa0a32fd5513525cddc8771 WatchSource:0}: Error finding container 6e7f4a8e2946f01d25622df7a97916ad52ae6f363aa0a32fd5513525cddc8771: Status 404 returned error can't find the container with id 6e7f4a8e2946f01d25622df7a97916ad52ae6f363aa0a32fd5513525cddc8771 Nov 03 16:38:44.230121 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:44.230082 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" event={"ID":"3a46652a-73e1-4a0e-95ab-668bb2126992","Type":"ContainerStarted","Data":"a4aeb09df74a8cf0c31f7922c2d50b39f28acb5c91a26f4c0851202d965260f3"} Nov 03 16:38:44.230518 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:44.230132 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" event={"ID":"3a46652a-73e1-4a0e-95ab-668bb2126992","Type":"ContainerStarted","Data":"6e7f4a8e2946f01d25622df7a97916ad52ae6f363aa0a32fd5513525cddc8771"} Nov 03 16:38:44.277495 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:44.277437 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podStartSLOduration=1.277423649 podStartE2EDuration="1.277423649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:44.275807578 +0000 UTC m=+1467.864807549" watchObservedRunningTime="2026-11-03 16:38:44.277423649 +0000 UTC m=+1467.866423619" Nov 03 16:38:44.797799 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:44.797763 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:45.299287 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.299253 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n"] Nov 03 16:38:45.303175 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.303153 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.341370 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.341334 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n"] Nov 03 16:38:45.395327 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.395295 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.395327 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.395326 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.395608 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.395353 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dxtw\" (UniqueName: \"kubernetes.io/projected/c7ef9905-f189-4f71-a6b9-10add14d0088-kube-api-access-4dxtw\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.395608 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.395506 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.395608 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.395551 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.395608 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.395571 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.395608 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.395587 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.395936 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.395752 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.395936 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.395829 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ef9905-f189-4f71-a6b9-10add14d0088-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.497436 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.497402 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.497601 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.497457 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.497601 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.497491 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.497601 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.497514 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.497601 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.497566 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.497932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.497614 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ef9905-f189-4f71-a6b9-10add14d0088-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.497932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.497820 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.497932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.497849 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.497932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.497887 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dxtw\" (UniqueName: \"kubernetes.io/projected/c7ef9905-f189-4f71-a6b9-10add14d0088-kube-api-access-4dxtw\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.498133 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.497966 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.498188 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.498171 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.498251 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.498187 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.498307 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.498238 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.498549 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.498531 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ef9905-f189-4f71-a6b9-10add14d0088-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.500086 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.500062 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.500388 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.500371 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.524305 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.524265 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.524438 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.524381 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dxtw\" (UniqueName: \"kubernetes.io/projected/c7ef9905-f189-4f71-a6b9-10add14d0088-kube-api-access-4dxtw\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.617071 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.616983 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:45.969277 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:38:45.969245 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ef9905_f189_4f71_a6b9_10add14d0088.slice/crio-564451b65d2b9cf07cd261cc8f6f947a61a0d4929970761fbea12cb897c71300 WatchSource:0}: Error finding container 564451b65d2b9cf07cd261cc8f6f947a61a0d4929970761fbea12cb897c71300: Status 404 returned error can't find the container with id 564451b65d2b9cf07cd261cc8f6f947a61a0d4929970761fbea12cb897c71300 Nov 03 16:38:45.969668 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.969643 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n"] Nov 03 16:38:45.971908 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.971866 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:38:45.972006 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.971952 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:38:45.972006 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:45.971998 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:38:46.243595 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:46.243560 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" event={"ID":"c7ef9905-f189-4f71-a6b9-10add14d0088","Type":"ContainerStarted","Data":"78c1d6a34d89dc9a909741647427717b9959c6cccfde0e8d7e16c1e9e9b3406f"} Nov 03 16:38:46.243819 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:46.243617 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" event={"ID":"c7ef9905-f189-4f71-a6b9-10add14d0088","Type":"ContainerStarted","Data":"564451b65d2b9cf07cd261cc8f6f947a61a0d4929970761fbea12cb897c71300"} Nov 03 16:38:46.300753 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:46.300665 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" podStartSLOduration=1.300644434 podStartE2EDuration="1.300644434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:38:46.299139451 +0000 UTC m=+1469.888139421" watchObservedRunningTime="2026-11-03 16:38:46.300644434 +0000 UTC m=+1469.889644405" Nov 03 16:38:46.622784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:46.620798 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:47.624392 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:47.624356 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:38:47.624815 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:47.624424 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" podUID="c7ef9905-f189-4f71-a6b9-10add14d0088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 16:38:48.617725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:48.617603 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:38:48.617725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:48.617666 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" podUID="c7ef9905-f189-4f71-a6b9-10add14d0088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 16:38:49.621723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:49.618066 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:38:49.621723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:49.618126 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" podUID="c7ef9905-f189-4f71-a6b9-10add14d0088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 16:38:50.620725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:50.620108 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:38:50.620725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:50.620172 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" podUID="c7ef9905-f189-4f71-a6b9-10add14d0088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 16:38:51.619474 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:51.619429 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:38:51.620118 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:51.620070 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" podUID="c7ef9905-f189-4f71-a6b9-10add14d0088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 16:38:52.622171 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:52.622131 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 16:38:52.622796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:52.622763 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" podUID="c7ef9905-f189-4f71-a6b9-10add14d0088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 16:38:53.787059 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:53.787025 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:53.787583 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:53.787561 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:53.789179 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:53.789153 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 16:38:53.798186 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:53.798158 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:53.810802 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:53.810772 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:38:54.307928 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:38:54.307897 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:39:00.129808 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:39:00.129725 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:39:05.387934 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.387897 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g"] Nov 03 16:39:05.391615 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.391592 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.407960 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.407934 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g"] Nov 03 16:39:05.411076 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.411055 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-491905\"/\"distinct-w1-491905-kserve-self-signed-certs\"" Nov 03 16:39:05.423391 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.423365 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:39:05.541022 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.540991 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-dshm\") pod \"distinct-w1-491905-kserve-69894cf86b-swc6g\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.541165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.541070 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-tmp-dir\") pod \"distinct-w1-491905-kserve-69894cf86b-swc6g\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.541165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.541143 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-model-cache\") pod \"distinct-w1-491905-kserve-69894cf86b-swc6g\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.541267 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.541170 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-722bm\" (UniqueName: \"kubernetes.io/projected/f1f96872-015d-4e32-adae-b8f9a552909e-kube-api-access-722bm\") pod \"distinct-w1-491905-kserve-69894cf86b-swc6g\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.541267 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.541209 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f96872-015d-4e32-adae-b8f9a552909e-tls-certs\") pod \"distinct-w1-491905-kserve-69894cf86b-swc6g\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.541468 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.541443 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-home\") pod \"distinct-w1-491905-kserve-69894cf86b-swc6g\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.642781 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.642678 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-tmp-dir\") pod \"distinct-w1-491905-kserve-69894cf86b-swc6g\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.642781 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.642758 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-model-cache\") pod \"distinct-w1-491905-kserve-69894cf86b-swc6g\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.643029 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.642786 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-722bm\" (UniqueName: \"kubernetes.io/projected/f1f96872-015d-4e32-adae-b8f9a552909e-kube-api-access-722bm\") pod \"distinct-w1-491905-kserve-69894cf86b-swc6g\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.643029 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.642819 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f96872-015d-4e32-adae-b8f9a552909e-tls-certs\") pod \"distinct-w1-491905-kserve-69894cf86b-swc6g\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.643029 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.642949 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-home\") pod \"distinct-w1-491905-kserve-69894cf86b-swc6g\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.643029 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.642998 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-dshm\") pod \"distinct-w1-491905-kserve-69894cf86b-swc6g\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.643237 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.643153 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-model-cache\") pod \"distinct-w1-491905-kserve-69894cf86b-swc6g\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.643237 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.643163 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-tmp-dir\") pod \"distinct-w1-491905-kserve-69894cf86b-swc6g\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.643492 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.643471 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-home\") pod \"distinct-w1-491905-kserve-69894cf86b-swc6g\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.645526 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.645505 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-dshm\") pod \"distinct-w1-491905-kserve-69894cf86b-swc6g\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.645692 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.645674 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f96872-015d-4e32-adae-b8f9a552909e-tls-certs\") pod \"distinct-w1-491905-kserve-69894cf86b-swc6g\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.667897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.667875 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-722bm\" (UniqueName: \"kubernetes.io/projected/f1f96872-015d-4e32-adae-b8f9a552909e-kube-api-access-722bm\") pod \"distinct-w1-491905-kserve-69894cf86b-swc6g\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:05.703432 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:05.703404 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:06.016885 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:06.016854 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g"] Nov 03 16:39:06.017032 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:39:06.017011 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1f96872_015d_4e32_adae_b8f9a552909e.slice/crio-4eb838c03171f72ef894ac2c9ea0ffca99de8116852333fe4abbb6efc7351b08 WatchSource:0}: Error finding container 4eb838c03171f72ef894ac2c9ea0ffca99de8116852333fe4abbb6efc7351b08: Status 404 returned error can't find the container with id 4eb838c03171f72ef894ac2c9ea0ffca99de8116852333fe4abbb6efc7351b08 Nov 03 16:39:06.377491 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:06.377397 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" event={"ID":"f1f96872-015d-4e32-adae-b8f9a552909e","Type":"ContainerStarted","Data":"70536f0162e8a5b210822a81676c3484ed0aaa8a6ddedd75db0564360e3f5e8d"} Nov 03 16:39:06.377491 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:06.377450 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" event={"ID":"f1f96872-015d-4e32-adae-b8f9a552909e","Type":"ContainerStarted","Data":"4eb838c03171f72ef894ac2c9ea0ffca99de8116852333fe4abbb6efc7351b08"} Nov 03 16:39:06.443766 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:06.443676 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podStartSLOduration=1.443656665 podStartE2EDuration="1.443656665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:06.439590386 +0000 UTC m=+1490.028590355" watchObservedRunningTime="2026-11-03 16:39:06.443656665 +0000 UTC m=+1490.032656635" Nov 03 16:39:06.703903 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:06.703790 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:15.399461 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.399426 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r"] Nov 03 16:39:15.404829 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.404804 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.451768 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.451738 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r"] Nov 03 16:39:15.527830 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.527800 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.528004 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.527839 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.528004 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.527933 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjtx\" (UniqueName: \"kubernetes.io/projected/d68c016e-91bf-4967-9d14-c6a0bea44dc4-kube-api-access-7kjtx\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.528004 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.527990 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.528112 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.528037 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.528112 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.528094 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.528191 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.528115 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.528191 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.528148 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.528258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.528221 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.629219 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.629184 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.629408 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.629341 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.629478 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.629421 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjtx\" (UniqueName: \"kubernetes.io/projected/d68c016e-91bf-4967-9d14-c6a0bea44dc4-kube-api-access-7kjtx\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.629586 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.629569 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.629654 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.629612 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.629654 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.629576 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.629828 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.629663 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.629828 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.629690 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.629828 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.629751 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.629828 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.629816 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.630036 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.629852 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.630036 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.629999 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.630172 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.630147 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.630326 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.630309 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.632071 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.632041 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.632258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.632239 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.656117 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.656058 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.656261 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.656244 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjtx\" (UniqueName: \"kubernetes.io/projected/d68c016e-91bf-4967-9d14-c6a0bea44dc4-kube-api-access-7kjtx\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:15.704207 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.704179 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:15.717654 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.717627 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:15.720827 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:15.720807 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:16.040293 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.040141 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r"] Nov 03 16:39:16.040422 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:39:16.040365 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68c016e_91bf_4967_9d14_c6a0bea44dc4.slice/crio-b379dd5966b0ca3ab1a99296b9e9e0bfe338ae84b1e24d5347cd6dcbb838ccc8 WatchSource:0}: Error finding container b379dd5966b0ca3ab1a99296b9e9e0bfe338ae84b1e24d5347cd6dcbb838ccc8: Status 404 returned error can't find the container with id b379dd5966b0ca3ab1a99296b9e9e0bfe338ae84b1e24d5347cd6dcbb838ccc8 Nov 03 16:39:16.042881 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.042847 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:39:16.042993 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.042927 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:39:16.042993 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.042956 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:39:16.296430 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.296354 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf"] Nov 03 16:39:16.301023 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.300990 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.331611 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.330096 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf"] Nov 03 16:39:16.338944 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.338902 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.339124 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.338991 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.339124 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.339034 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.339124 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.339081 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.339124 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.339123 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.339349 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.339157 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbq58\" (UniqueName: \"kubernetes.io/projected/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-kube-api-access-dbq58\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.339349 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.339189 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.339349 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.339328 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.339510 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.339353 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.441765 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.441728 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.442235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.441786 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.442235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.441824 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbq58\" (UniqueName: \"kubernetes.io/projected/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-kube-api-access-dbq58\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.442235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.441854 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.442235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.441989 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.442235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.442012 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.442235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.442098 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.442235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.442165 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.442235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.442169 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.442235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.442206 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.442726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.442473 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.442726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.442544 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.442726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.442687 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.442903 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.442812 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" event={"ID":"d68c016e-91bf-4967-9d14-c6a0bea44dc4","Type":"ContainerStarted","Data":"4e1c0285516289527713c98153cfbd396f4e438b85cd4fe228cde0adeeafe9ac"} Nov 03 16:39:16.442903 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.442862 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" event={"ID":"d68c016e-91bf-4967-9d14-c6a0bea44dc4","Type":"ContainerStarted","Data":"b379dd5966b0ca3ab1a99296b9e9e0bfe338ae84b1e24d5347cd6dcbb838ccc8"} Nov 03 16:39:16.443378 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.443346 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.446147 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.446120 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.446296 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.446216 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.455747 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.455728 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:16.467320 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.467283 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbq58\" (UniqueName: \"kubernetes.io/projected/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-kube-api-access-dbq58\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.468773 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.468753 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.494581 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.494538 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" podStartSLOduration=1.4945240659999999 podStartE2EDuration="1.494524066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:16.493288551 +0000 UTC m=+1500.082288521" watchObservedRunningTime="2026-11-03 16:39:16.494524066 +0000 UTC m=+1500.083524065" Nov 03 16:39:16.619527 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.619448 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:16.721017 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.720969 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:16.748059 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.748027 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:39:16.748320 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.748300 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:39:16.956287 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.956259 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf"] Nov 03 16:39:16.956957 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:39:16.956929 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd281375d_bb6c_4ac6_b2fb_8bd4cb476178.slice/crio-758978b20617dd13890de58bbdf059f98c2ac01bac9b50b7dd6e71cf0934008b WatchSource:0}: Error finding container 758978b20617dd13890de58bbdf059f98c2ac01bac9b50b7dd6e71cf0934008b: Status 404 returned error can't find the container with id 758978b20617dd13890de58bbdf059f98c2ac01bac9b50b7dd6e71cf0934008b Nov 03 16:39:16.959552 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.959507 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:39:16.959658 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.959595 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:39:16.959658 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:16.959636 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:39:17.192503 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.192426 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84"] Nov 03 16:39:17.196620 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.196595 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.231366 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.231332 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84"] Nov 03 16:39:17.256436 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.256351 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2bd4293-1c17-455d-9237-38edc12c9890-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.256436 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.256429 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2bd4293-1c17-455d-9237-38edc12c9890-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.256904 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.256541 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.256904 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.256613 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.256904 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.256660 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.256904 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.256727 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxw76\" (UniqueName: \"kubernetes.io/projected/f2bd4293-1c17-455d-9237-38edc12c9890-kube-api-access-bxw76\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.256904 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.256756 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.256904 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.256811 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.256904 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.256836 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2bd4293-1c17-455d-9237-38edc12c9890-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.358575 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.358538 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.358773 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.358599 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.358773 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.358646 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxw76\" (UniqueName: \"kubernetes.io/projected/f2bd4293-1c17-455d-9237-38edc12c9890-kube-api-access-bxw76\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.358773 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.358676 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.358960 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.358781 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.358960 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.358797 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2bd4293-1c17-455d-9237-38edc12c9890-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.358960 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.358858 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2bd4293-1c17-455d-9237-38edc12c9890-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.358960 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.358907 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2bd4293-1c17-455d-9237-38edc12c9890-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.359154 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.359010 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.359522 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.359491 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.359654 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.359622 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.359768 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.359747 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.359927 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.359903 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.360003 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.359912 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2bd4293-1c17-455d-9237-38edc12c9890-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.361937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.361859 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.362150 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.362131 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2bd4293-1c17-455d-9237-38edc12c9890-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.385785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.385755 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxw76\" (UniqueName: \"kubernetes.io/projected/f2bd4293-1c17-455d-9237-38edc12c9890-kube-api-access-bxw76\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.387105 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.387086 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2bd4293-1c17-455d-9237-38edc12c9890-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.450796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.450650 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" event={"ID":"d281375d-bb6c-4ac6-b2fb-8bd4cb476178","Type":"ContainerStarted","Data":"58d3051d9350dc8cc3494e1eeb3fc792c8029a29bcf41f3d3614d697e83ffd7e"} Nov 03 16:39:17.450796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.450733 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" event={"ID":"d281375d-bb6c-4ac6-b2fb-8bd4cb476178","Type":"ContainerStarted","Data":"758978b20617dd13890de58bbdf059f98c2ac01bac9b50b7dd6e71cf0934008b"} Nov 03 16:39:17.522731 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.516174 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" podStartSLOduration=1.5161543960000001 podStartE2EDuration="1.516154396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:17.514192538 +0000 UTC m=+1501.103192509" watchObservedRunningTime="2026-11-03 16:39:17.516154396 +0000 UTC m=+1501.105154369" Nov 03 16:39:17.538201 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.538153 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:17.636752 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.631890 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:17.730946 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.730852 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:17.730946 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.730934 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" podUID="d68c016e-91bf-4967-9d14-c6a0bea44dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 16:39:17.863062 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.862845 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84"] Nov 03 16:39:17.864162 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:39:17.864135 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2bd4293_1c17_455d_9237_38edc12c9890.slice/crio-d39a07332468bcbf0fb3565ce4a28061bbf4887b0728a9333bc60a1dd8263468 WatchSource:0}: Error finding container d39a07332468bcbf0fb3565ce4a28061bbf4887b0728a9333bc60a1dd8263468: Status 404 returned error can't find the container with id d39a07332468bcbf0fb3565ce4a28061bbf4887b0728a9333bc60a1dd8263468 Nov 03 16:39:17.866315 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.866284 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:39:17.866404 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.866344 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:39:17.866404 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:17.866376 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:39:18.464822 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:18.464786 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" event={"ID":"f2bd4293-1c17-455d-9237-38edc12c9890","Type":"ContainerStarted","Data":"9088ac672a34d90115e6063e35401fc27e3e6e4e5ef995344bedfe610c88a293"} Nov 03 16:39:18.465377 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:18.465357 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" event={"ID":"f2bd4293-1c17-455d-9237-38edc12c9890","Type":"ContainerStarted","Data":"d39a07332468bcbf0fb3565ce4a28061bbf4887b0728a9333bc60a1dd8263468"} Nov 03 16:39:18.546777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:18.546650 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:18.555867 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:18.555832 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:39:18.556166 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:18.556120 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" podUID="f2bd4293-1c17-455d-9237-38edc12c9890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 16:39:18.631735 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:18.629888 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:39:18.631735 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:18.629948 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" podUID="d281375d-bb6c-4ac6-b2fb-8bd4cb476178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 16:39:18.746486 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:18.742243 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:18.746486 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:18.742310 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" podUID="d68c016e-91bf-4967-9d14-c6a0bea44dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 16:39:19.552734 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:19.547178 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:39:19.552734 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:19.547247 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" podUID="f2bd4293-1c17-455d-9237-38edc12c9890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 16:39:19.622724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:19.622591 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:39:19.622724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:19.622722 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" podUID="d281375d-bb6c-4ac6-b2fb-8bd4cb476178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 16:39:19.723724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:19.722172 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:19.723724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:19.722237 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" podUID="d68c016e-91bf-4967-9d14-c6a0bea44dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 16:39:20.543723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:20.543117 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:39:20.543723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:20.543178 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" podUID="f2bd4293-1c17-455d-9237-38edc12c9890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 16:39:20.799782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:20.701280 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:39:20.799782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:20.701348 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" podUID="d281375d-bb6c-4ac6-b2fb-8bd4cb476178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 16:39:20.799782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:20.721913 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:20.799782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:20.721964 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" podUID="d68c016e-91bf-4967-9d14-c6a0bea44dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 16:39:20.799782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:20.785826 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:21.042727 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:21.031048 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:21.043462 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:21.043410 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:21.305723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:21.304984 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:21.471724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:21.450781 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 16:39:21.557339 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:21.557252 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:39:21.557597 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:21.557565 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" podUID="f2bd4293-1c17-455d-9237-38edc12c9890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 16:39:21.712732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:21.708829 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:39:21.712732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:21.708901 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" podUID="d281375d-bb6c-4ac6-b2fb-8bd4cb476178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 16:39:21.761113 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:21.761067 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:21.761440 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:21.761389 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" podUID="d68c016e-91bf-4967-9d14-c6a0bea44dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 16:39:21.787633 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:21.787583 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:21.789276 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:21.789253 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:39:21.876831 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:21.876717 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 16:39:21.986743 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:21.974021 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" podStartSLOduration=4.974000172 podStartE2EDuration="4.974000172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:18.546612621 +0000 UTC m=+1502.135612590" watchObservedRunningTime="2026-11-03 16:39:21.974000172 +0000 UTC m=+1505.563000142" Nov 03 16:39:22.074760 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.067661 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 16:39:22.074760 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.068071 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:22.074760 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.068399 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:39:22.074760 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.068458 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:39:22.318748 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.313805 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 16:39:22.318748 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.317867 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:39:22.461861 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.461818 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 16:39:22.462893 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.462869 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:22.603782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.583945 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:39:22.603782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.584027 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" podUID="8c8da48a-94af-4d09-aef7-21c1a69092c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:22.710724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.707268 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:39:22.714970 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.714931 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" podUID="d281375d-bb6c-4ac6-b2fb-8bd4cb476178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 16:39:22.742793 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.729931 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:39:22.743048 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.743016 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" podUID="f2bd4293-1c17-455d-9237-38edc12c9890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 16:39:22.747829 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.747800 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:22.760736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.748004 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" podUID="d68c016e-91bf-4967-9d14-c6a0bea44dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 16:39:22.833725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.805883 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:22.940785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.923819 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:22.940785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:22.924300 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:39:23.175940 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:23.175881 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:23.179803 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:23.179755 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:23.427825 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:23.427763 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:23.599204 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:23.599173 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:39:23.667724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:23.652973 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:39:23.667724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:23.653035 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" podUID="f2bd4293-1c17-455d-9237-38edc12c9890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 16:39:23.687749 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:23.687641 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:39:23.688031 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:23.687987 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" podUID="d281375d-bb6c-4ac6-b2fb-8bd4cb476178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 16:39:23.705147 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:23.705103 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:23.972805 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:23.884796 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 16:39:23.986528 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:23.986488 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:39:24.067767 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:24.067723 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:24.068832 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:24.068800 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" podUID="d68c016e-91bf-4967-9d14-c6a0bea44dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 16:39:24.151658 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:24.151630 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:24.179922 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:24.179879 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:24.191387 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:24.191358 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:39:24.284806 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:24.284778 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:39:24.290532 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:24.290506 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:39:24.542749 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:24.542621 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:39:24.552805 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:24.542971 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" podUID="f2bd4293-1c17-455d-9237-38edc12c9890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 16:39:24.625726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:24.625237 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:39:24.625726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:24.625312 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" podUID="d281375d-bb6c-4ac6-b2fb-8bd4cb476178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 16:39:24.769723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:24.751827 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 16:39:24.769723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:24.751896 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" podUID="d68c016e-91bf-4967-9d14-c6a0bea44dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 16:39:25.539254 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:25.539215 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:39:25.539946 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:25.539913 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" podUID="f2bd4293-1c17-455d-9237-38edc12c9890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 16:39:25.624131 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:25.624090 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 16:39:25.624420 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:25.624392 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" podUID="d281375d-bb6c-4ac6-b2fb-8bd4cb476178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 16:39:26.030302 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:26.030275 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:26.031498 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:26.031469 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:26.032364 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:26.032343 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 16:39:26.539091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:26.539054 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 16:39:26.539289 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:26.539116 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" podUID="f2bd4293-1c17-455d-9237-38edc12c9890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 16:39:27.660916 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:27.660866 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:39:27.661461 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:27.660954 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" podUID="d281375d-bb6c-4ac6-b2fb-8bd4cb476178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:27.842668 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:27.842637 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:27.843085 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:27.843063 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:27.843970 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:27.843950 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 16:39:28.605240 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:28.605205 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:39:28.605531 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:28.605502 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" podUID="f2bd4293-1c17-455d-9237-38edc12c9890" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:28.679594 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:28.679566 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:28.680258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:28.680239 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:28.681282 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:28.681258 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 16:39:28.966557 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:28.966473 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z"] Nov 03 16:39:28.973824 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:28.973794 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:28.996041 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:28.996010 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-71d7b4\"/\"distinct-2-w0-71d7b4-kserve-self-signed-certs\"" Nov 03 16:39:28.998552 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:28.998532 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z"] Nov 03 16:39:29.060498 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.060452 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-tls-certs\") pod \"distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.060685 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.060589 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-model-cache\") pod \"distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.060685 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.060658 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-home\") pod \"distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.060848 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.060763 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-dshm\") pod \"distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.060848 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.060822 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl4td\" (UniqueName: \"kubernetes.io/projected/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-kube-api-access-xl4td\") pod \"distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.060937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.060914 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-tmp-dir\") pod \"distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.177665 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.172325 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-tls-certs\") pod \"distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.177665 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.172441 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-model-cache\") pod \"distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.177665 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.172522 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-home\") pod \"distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.177665 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.172562 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-dshm\") pod \"distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.177665 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.172717 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl4td\" (UniqueName: \"kubernetes.io/projected/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-kube-api-access-xl4td\") pod \"distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.177665 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.172780 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-tmp-dir\") pod \"distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.177665 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.173294 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-tmp-dir\") pod \"distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.181724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.180421 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-model-cache\") pod \"distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.181724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.180872 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-home\") pod \"distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.221904 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.218007 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-dshm\") pod \"distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.221904 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.218776 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-tls-certs\") pod \"distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.249717 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.237862 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl4td\" (UniqueName: \"kubernetes.io/projected/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-kube-api-access-xl4td\") pod \"distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.292727 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.292612 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:29.621004 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:39:29.620976 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a3bbeb6_89f0_49af_9b1e_f5c3bf03ccdd.slice/crio-7eeb586f9ee4b99e6b4a5e589d0b51cbacd1eb63a512d0e978bc6c6d24603a4b WatchSource:0}: Error finding container 7eeb586f9ee4b99e6b4a5e589d0b51cbacd1eb63a512d0e978bc6c6d24603a4b: Status 404 returned error can't find the container with id 7eeb586f9ee4b99e6b4a5e589d0b51cbacd1eb63a512d0e978bc6c6d24603a4b Nov 03 16:39:29.621137 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:29.621116 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z"] Nov 03 16:39:30.130641 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:39:30.130610 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:39:30.139421 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:30.139391 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" event={"ID":"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd","Type":"ContainerStarted","Data":"3516a3a7c217beee742963a40c8c765115762d317ce1ded8cbf73c5ef0c8e638"} Nov 03 16:39:30.139651 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:30.139432 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" event={"ID":"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd","Type":"ContainerStarted","Data":"7eeb586f9ee4b99e6b4a5e589d0b51cbacd1eb63a512d0e978bc6c6d24603a4b"} Nov 03 16:39:30.227793 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:30.227737 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" podStartSLOduration=2.227719932 podStartE2EDuration="2.227719932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:30.224846859 +0000 UTC m=+1513.813846827" watchObservedRunningTime="2026-11-03 16:39:30.227719932 +0000 UTC m=+1513.816719898" Nov 03 16:39:30.293831 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:30.293759 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:33.102882 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:33.102847 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:39:36.236404 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.236374 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv"] Nov 03 16:39:36.240594 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.240574 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.268369 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.268338 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv"] Nov 03 16:39:36.336217 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.336184 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.336389 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.336227 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.336473 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.336386 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.336531 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.336499 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.336584 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.336550 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt2qv\" (UniqueName: \"kubernetes.io/projected/4028bf00-f9f7-44ce-9de1-4ffe922e640b-kube-api-access-jt2qv\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.336628 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.336596 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.336686 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.336667 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.336781 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.336767 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.336817 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.336800 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.437418 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.437387 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.437418 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.437422 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.437617 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.437488 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.437617 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.437563 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.437617 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.437592 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt2qv\" (UniqueName: \"kubernetes.io/projected/4028bf00-f9f7-44ce-9de1-4ffe922e640b-kube-api-access-jt2qv\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.437798 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.437633 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.437798 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.437658 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.437912 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.437859 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.437970 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.437907 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.438029 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.437904 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.438080 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.438011 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.438256 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.438231 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.438329 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.438257 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.438404 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.438385 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.440457 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.440435 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.440874 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.440851 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.462541 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.462516 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt2qv\" (UniqueName: \"kubernetes.io/projected/4028bf00-f9f7-44ce-9de1-4ffe922e640b-kube-api-access-jt2qv\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.462814 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.462794 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.554313 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.554273 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:36.879386 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:39:36.879356 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4028bf00_f9f7_44ce_9de1_4ffe922e640b.slice/crio-7bd4a1cb085a67da20bc7fd03b4a8b344f571c0a9c83ffdb5d39cd7f73e6927c WatchSource:0}: Error finding container 7bd4a1cb085a67da20bc7fd03b4a8b344f571c0a9c83ffdb5d39cd7f73e6927c: Status 404 returned error can't find the container with id 7bd4a1cb085a67da20bc7fd03b4a8b344f571c0a9c83ffdb5d39cd7f73e6927c Nov 03 16:39:36.879493 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.879472 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv"] Nov 03 16:39:36.881884 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.881853 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:39:36.881946 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.881917 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:39:36.881987 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:36.881946 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:39:37.196418 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:37.196333 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" event={"ID":"4028bf00-f9f7-44ce-9de1-4ffe922e640b","Type":"ContainerStarted","Data":"75737fefb5726f13651957ace2a3029d8aacd68ea97d0951445dcf4d15888e63"} Nov 03 16:39:37.196418 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:37.196399 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" event={"ID":"4028bf00-f9f7-44ce-9de1-4ffe922e640b","Type":"ContainerStarted","Data":"7bd4a1cb085a67da20bc7fd03b4a8b344f571c0a9c83ffdb5d39cd7f73e6927c"} Nov 03 16:39:37.252323 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:37.252271 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" podStartSLOduration=1.252256659 podStartE2EDuration="1.252256659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:37.249455729 +0000 UTC m=+1520.838455691" watchObservedRunningTime="2026-11-03 16:39:37.252256659 +0000 UTC m=+1520.841256627" Nov 03 16:39:37.555099 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:37.555067 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:37.556777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:37.556757 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:39:37.556829 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:37.556806 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" podUID="4028bf00-f9f7-44ce-9de1-4ffe922e640b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 16:39:38.555255 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:38.555227 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:39:38.555632 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:38.555276 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" podUID="4028bf00-f9f7-44ce-9de1-4ffe922e640b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 16:39:39.293474 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:39.293439 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:39.334544 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:39.334515 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:39.555002 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:39.554909 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:39:39.555002 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:39.554978 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" podUID="4028bf00-f9f7-44ce-9de1-4ffe922e640b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 16:39:40.258512 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:40.258483 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:40.555421 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:40.555335 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:39:40.555421 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:40.555393 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" podUID="4028bf00-f9f7-44ce-9de1-4ffe922e640b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 16:39:41.555027 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:41.554989 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:39:41.555480 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:41.555076 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" podUID="4028bf00-f9f7-44ce-9de1-4ffe922e640b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 16:39:42.555322 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:42.555292 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:39:42.555684 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:42.555351 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" podUID="4028bf00-f9f7-44ce-9de1-4ffe922e640b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 16:39:43.555391 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:43.555358 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 16:39:43.555847 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:43.555427 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" podUID="4028bf00-f9f7-44ce-9de1-4ffe922e640b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 16:39:44.565778 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:44.565751 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:44.566329 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:44.566031 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:44.567073 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:44.567057 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 16:39:46.173289 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.173258 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4"] Nov 03 16:39:46.179401 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.179373 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.201525 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.201492 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4"] Nov 03 16:39:46.211562 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.211523 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-491905\"/\"distinct-2-w1-491905-kserve-self-signed-certs\"" Nov 03 16:39:46.323925 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.323892 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-home\") pod \"distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.323925 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.323925 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-tls-certs\") pod \"distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.324145 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.324029 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-dshm\") pod \"distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.324145 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.324071 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-model-cache\") pod \"distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.324226 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.324205 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr7rn\" (UniqueName: \"kubernetes.io/projected/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-kube-api-access-sr7rn\") pod \"distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.324269 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.324242 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-tmp-dir\") pod \"distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.426668 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.426565 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-dshm\") pod \"distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.444360 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.444329 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-model-cache\") pod \"distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.445138 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.445115 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr7rn\" (UniqueName: \"kubernetes.io/projected/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-kube-api-access-sr7rn\") pod \"distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.445875 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.445853 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-tmp-dir\") pod \"distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.446178 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.446157 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-home\") pod \"distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.446309 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.446292 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-tls-certs\") pod \"distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.457906 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.444913 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-model-cache\") pod \"distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.457906 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.457538 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-tmp-dir\") pod \"distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.457906 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.457858 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-home\") pod \"distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.479795 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.479534 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-tls-certs\") pod \"distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.480246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.479628 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-dshm\") pod \"distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.493749 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.493225 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr7rn\" (UniqueName: \"kubernetes.io/projected/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-kube-api-access-sr7rn\") pod \"distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.495577 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.495119 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:46.846366 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:46.846338 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4"] Nov 03 16:39:46.847520 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:39:46.847494 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f4b9526_8df4_4d0f_a9bc_34824126f4ca.slice/crio-ba1030c7a530eb645a0ce631feec0a53bd68290d1fcadb3cb2f6785f7550d28f WatchSource:0}: Error finding container ba1030c7a530eb645a0ce631feec0a53bd68290d1fcadb3cb2f6785f7550d28f: Status 404 returned error can't find the container with id ba1030c7a530eb645a0ce631feec0a53bd68290d1fcadb3cb2f6785f7550d28f Nov 03 16:39:47.289242 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.289208 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" event={"ID":"2f4b9526-8df4-4d0f-a9bc-34824126f4ca","Type":"ContainerStarted","Data":"d0b5a2d2850a9588f753a26249a1fdf03865734f7b7ddbd81950dcff11502b0d"} Nov 03 16:39:47.289242 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.289248 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" event={"ID":"2f4b9526-8df4-4d0f-a9bc-34824126f4ca","Type":"ContainerStarted","Data":"ba1030c7a530eb645a0ce631feec0a53bd68290d1fcadb3cb2f6785f7550d28f"} Nov 03 16:39:47.300620 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.300592 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9"] Nov 03 16:39:47.304812 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.304793 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.334899 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.334866 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9"] Nov 03 16:39:47.340519 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.339656 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" podStartSLOduration=1.339638115 podStartE2EDuration="1.339638115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:47.33895539 +0000 UTC m=+1530.927955359" watchObservedRunningTime="2026-11-03 16:39:47.339638115 +0000 UTC m=+1530.928638085" Nov 03 16:39:47.498659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.494749 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.498659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.494830 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.498659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.494951 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1971c181-22d1-4f72-a032-f65f2a6b765f-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.498659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.495030 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgb4x\" (UniqueName: \"kubernetes.io/projected/1971c181-22d1-4f72-a032-f65f2a6b765f-kube-api-access-qgb4x\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.498659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.495128 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.498659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.495184 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.498659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.495211 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.498659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.495241 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.498659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.495274 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.498659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.496027 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:47.596735 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.596619 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.596735 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.596654 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.596735 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.596684 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.597035 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.596778 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.597035 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.596890 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.597035 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.597000 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.597192 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.597086 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.597192 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.597089 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1971c181-22d1-4f72-a032-f65f2a6b765f-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.597192 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.597149 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.597323 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.597204 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgb4x\" (UniqueName: \"kubernetes.io/projected/1971c181-22d1-4f72-a032-f65f2a6b765f-kube-api-access-qgb4x\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.597323 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.597258 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.597323 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.597288 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.597500 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.597482 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.597835 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.597752 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1971c181-22d1-4f72-a032-f65f2a6b765f-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.599469 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.599444 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.599817 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.599797 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.622050 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.622020 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.622162 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.622097 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgb4x\" (UniqueName: \"kubernetes.io/projected/1971c181-22d1-4f72-a032-f65f2a6b765f-kube-api-access-qgb4x\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:47.920740 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:47.920637 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:48.240167 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:48.240135 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9"] Nov 03 16:39:48.240406 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:39:48.240376 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1971c181_22d1_4f72_a032_f65f2a6b765f.slice/crio-f482fc496cd6dd564222becd22a38e1d0b0edfaa1a6f0a45b32a50d1c5895ca6 WatchSource:0}: Error finding container f482fc496cd6dd564222becd22a38e1d0b0edfaa1a6f0a45b32a50d1c5895ca6: Status 404 returned error can't find the container with id f482fc496cd6dd564222becd22a38e1d0b0edfaa1a6f0a45b32a50d1c5895ca6 Nov 03 16:39:48.242901 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:48.242871 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:39:48.242969 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:48.242936 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:39:48.243007 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:48.242967 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:39:48.297595 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:48.297567 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" event={"ID":"1971c181-22d1-4f72-a032-f65f2a6b765f","Type":"ContainerStarted","Data":"f482fc496cd6dd564222becd22a38e1d0b0edfaa1a6f0a45b32a50d1c5895ca6"} Nov 03 16:39:49.330040 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:49.330007 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" event={"ID":"1971c181-22d1-4f72-a032-f65f2a6b765f","Type":"ContainerStarted","Data":"6419d9b304b49cef0c84c4ec3ea7f471ada98fdb8dbf9f4e023d8feb38f7c059"} Nov 03 16:39:49.394036 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:49.393977 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" podStartSLOduration=2.39396125 podStartE2EDuration="2.39396125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:39:49.39124031 +0000 UTC m=+1532.980240283" watchObservedRunningTime="2026-11-03 16:39:49.39396125 +0000 UTC m=+1532.982961218" Nov 03 16:39:49.923022 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:49.922908 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:39:49.924710 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:49.924640 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:39:49.924853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:49.924771 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" podUID="1971c181-22d1-4f72-a032-f65f2a6b765f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 16:39:50.921720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:50.921134 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:39:50.921720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:50.921202 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" podUID="1971c181-22d1-4f72-a032-f65f2a6b765f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 16:39:51.923726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:51.922131 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:39:51.923726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:51.922199 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" podUID="1971c181-22d1-4f72-a032-f65f2a6b765f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 16:39:52.929724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:52.928320 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:39:52.929724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:52.928382 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" podUID="1971c181-22d1-4f72-a032-f65f2a6b765f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 16:39:53.924207 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:53.924159 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:39:53.924545 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:53.924484 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" podUID="1971c181-22d1-4f72-a032-f65f2a6b765f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 16:39:54.921180 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:54.921138 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:39:54.923181 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:54.923147 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" podUID="1971c181-22d1-4f72-a032-f65f2a6b765f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 16:39:55.153734 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:55.152862 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:55.195302 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:55.195205 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 16:39:55.287840 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:55.287793 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:56.199725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:56.109305 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 16:39:56.199725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:56.156887 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:56.199725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:56.157420 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:56.236725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:56.213808 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 16:39:56.236725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:56.214265 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 16:39:56.236725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:56.221932 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:56.236725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:56.232902 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:39:56.261725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:56.250152 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:56.310746 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:56.307818 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:56.310746 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:56.308290 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:39:56.611759 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:56.608574 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:39:56.689582 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:56.608678 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" podUID="1971c181-22d1-4f72-a032-f65f2a6b765f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 16:39:56.739504 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:56.739448 4947 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:56.885233 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:56.885110 4947 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 03 16:39:56.913763 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:56.766690 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:39:57.120726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.117564 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 16:39:57.161042 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.160951 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 16:39:57.263825 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.263784 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" podUID="1971c181-22d1-4f72-a032-f65f2a6b765f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 16:39:57.378091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.239016 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 03 16:39:57.378091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.263836 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:57.378091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.180420 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:57.378091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.179821 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:39:57.378091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.364309 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" podUID="d68c016e-91bf-4967-9d14-c6a0bea44dc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:57.378091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.324011 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:57.378091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.320898 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 16:39:57.378091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.324057 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:57.387914 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.387878 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:39:57.388227 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.388197 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" podUID="ee442df4-03ea-4f56-9b1a-a31e84d2c861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:57.700724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.696955 4947 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" podUID="2f4b9526-8df4-4d0f-a9bc-34824126f4ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded" Nov 03 16:39:57.768717 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.764931 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:39:57.768717 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.765103 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:39:57.768717 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.765170 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:57.768717 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.765224 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:39:57.768717 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.765262 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:39:57.768717 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:57.765309 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:39:58.143028 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:58.142998 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:39:58.199056 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:58.199019 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:39:58.261625 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:58.261581 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:39:58.287793 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:58.287764 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:39:58.387400 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:58.383828 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:39:58.484043 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:58.479523 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:39:58.970723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:58.968544 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:39:58.970723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:39:58.968604 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" podUID="1971c181-22d1-4f72-a032-f65f2a6b765f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:00.010923 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:00.010879 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:00.011440 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:00.010955 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" podUID="1971c181-22d1-4f72-a032-f65f2a6b765f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:00.135311 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:40:00.135234 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:40:00.320153 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:00.320076 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:40:00.320368 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:00.320347 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:40:00.321300 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:00.321277 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:40:04.899370 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:04.899336 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:40:06.541803 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:06.541770 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:40:06.594147 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:06.594115 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:40:07.003403 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:07.003246 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:07.147724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:07.146964 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:07.265876 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:07.265778 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:07.384855 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:07.384803 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:07.411108 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:07.385103 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:07.414052 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:07.413052 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:07.436725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:07.413081 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 03 16:40:08.032894 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:08.032845 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:08.058744 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:08.041219 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:40:08.268801 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:08.215767 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:08.268801 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:08.216542 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:40:08.301827 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:08.269654 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:08.301827 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:08.270403 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:40:08.476727 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:08.451428 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:08.477257 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:08.468106 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:08.478093 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:08.478069 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:40:08.478294 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:08.478279 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:40:08.493724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:08.469011 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:08.494260 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:08.469057 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:08.519722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:08.494731 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:40:08.520049 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:08.520028 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:40:08.633512 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:08.575790 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" podUID="2f4b9526-8df4-4d0f-a9bc-34824126f4ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 16:40:09.072725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:09.064675 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 16:40:09.237753 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:09.229460 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.261495 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:09.261450 4947 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.289775 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:09.289729 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.487068 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:09.486973 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.502067 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:09.487942 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.527936 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:09.504320 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.535486 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:09.504354 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 16:40:09.716433 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:09.696903 4947 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": context deadline exceeded" Nov 03 16:40:09.866551 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:09.866516 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:09.869090 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:09.869052 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" podUID="c7ef9905-f189-4f71-a6b9-10add14d0088" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:09.904738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:09.903616 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:40:09.948377 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:09.948352 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:40:09.957864 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:09.957840 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:40:09.974310 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:09.974282 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:40:09.981332 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:09.978112 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:40:10.003413 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:10.003281 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:40:10.036206 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:10.035316 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:40:21.411776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.411736 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq"] Nov 03 16:40:21.459541 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.459506 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq"] Nov 03 16:40:21.459900 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.459879 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.665785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.595451 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.665785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.595507 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.665785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.595541 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38262c13-420b-43af-bb5e-1cb3c62e44b0-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.665785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.595572 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.665785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.595600 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.665785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.595638 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.665785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.595783 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxccp\" (UniqueName: \"kubernetes.io/projected/38262c13-420b-43af-bb5e-1cb3c62e44b0-kube-api-access-rxccp\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.665785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.595862 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.665785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.595924 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.744736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.701944 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxccp\" (UniqueName: \"kubernetes.io/projected/38262c13-420b-43af-bb5e-1cb3c62e44b0-kube-api-access-rxccp\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.744736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.702040 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.744736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.702132 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.744736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.702240 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.744736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.702272 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.744736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.702304 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38262c13-420b-43af-bb5e-1cb3c62e44b0-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.744736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.702334 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.744736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.702364 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.744736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.702409 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.800727 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.750867 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38262c13-420b-43af-bb5e-1cb3c62e44b0-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.800727 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.751591 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.800727 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.752359 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.800727 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.752672 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.842726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.802519 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.907726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.885336 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.907726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.896323 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.907726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.897005 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxccp\" (UniqueName: \"kubernetes.io/projected/38262c13-420b-43af-bb5e-1cb3c62e44b0-kube-api-access-rxccp\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.908046 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.907801 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:21.924781 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.924051 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:21.956726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.955861 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:21.980352 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:21.979938 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 16:40:22.029854 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:22.029806 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:22.049891 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:22.046799 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:22.049891 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:22.049166 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:22.050411 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:22.050387 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:22.244644 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:22.243455 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:22.755740 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:22.728103 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" podUID="2f4b9526-8df4-4d0f-a9bc-34824126f4ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:22.809576 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:22.793689 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:40:22.809576 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:22.793786 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" podUID="8c8da48a-94af-4d09-aef7-21c1a69092c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:22.947726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:22.917240 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:22.948789 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:22.948758 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:40:22.966856 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:22.966808 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:22.994899 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:22.994866 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:40:22.996063 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:22.996029 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 16:40:22.996564 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:22.996543 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:40:23.086780 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:23.055288 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:23.088220 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:23.087083 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 16:40:23.125893 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:23.087128 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:23.125893 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:23.089262 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:40:23.125893 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:23.089341 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:40:23.518260 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:23.435673 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:23.518260 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:23.456213 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:40:23.518260 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:23.494823 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:40:23.919253 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:23.906467 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" podUID="2f4b9526-8df4-4d0f-a9bc-34824126f4ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 16:40:23.931192 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:23.929014 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:40:23.963494 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:23.963466 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 16:40:23.979810 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:23.978367 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:40:24.004005 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:23.994821 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.007608 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:24.006597 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.009141 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:24.009102 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:40:24.041333 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:24.041293 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:40:24.041826 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:24.041793 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:40:24.042571 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:24.011894 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:40:24.095026 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:24.089445 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:40:24.095394 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:24.095375 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:40:24.097873 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:24.097848 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq"] Nov 03 16:40:24.122725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:24.122685 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:40:24.250299 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:24.250268 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:40:24.250823 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:24.250802 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:40:24.267859 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:24.267837 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:40:25.153083 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:25.153046 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" event={"ID":"38262c13-420b-43af-bb5e-1cb3c62e44b0","Type":"ContainerStarted","Data":"35845104c931053556b35962ec1885befcf0bdb46424a452dbeb7efa97837bf3"} Nov 03 16:40:25.153579 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:25.153114 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" event={"ID":"38262c13-420b-43af-bb5e-1cb3c62e44b0","Type":"ContainerStarted","Data":"208a99156ffbf8f1fa1ade0dc9baa4075e22911053eaf6d264952a1b5485c1a5"} Nov 03 16:40:26.051365 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:26.051328 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:26.052544 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:26.052518 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 16:40:26.052671 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:26.052570 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" podUID="38262c13-420b-43af-bb5e-1cb3c62e44b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 16:40:27.051912 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:27.051877 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 16:40:27.052338 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:27.051933 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" podUID="38262c13-420b-43af-bb5e-1cb3c62e44b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 16:40:28.051640 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:28.051601 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 16:40:28.051831 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:28.051663 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" podUID="38262c13-420b-43af-bb5e-1cb3c62e44b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 16:40:29.059133 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.059095 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 16:40:29.059781 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.059745 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" podUID="38262c13-420b-43af-bb5e-1cb3c62e44b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 16:40:29.634269 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.634217 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" podStartSLOduration=8.634202747 podStartE2EDuration="8.634202747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:25.225144948 +0000 UTC m=+1568.814144919" watchObservedRunningTime="2026-11-03 16:40:29.634202747 +0000 UTC m=+1573.223202718" Nov 03 16:40:29.636615 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.636591 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw"] Nov 03 16:40:29.640849 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.640833 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.650361 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.650342 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf"] Nov 03 16:40:29.656094 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.656076 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.679915 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.679886 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw"] Nov 03 16:40:29.689613 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.689589 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf"] Nov 03 16:40:29.747049 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.747014 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.747207 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.747075 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.747207 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.747110 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.747207 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.747174 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.747312 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.747249 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.747312 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.747276 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.747394 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.747344 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr2md\" (UniqueName: \"kubernetes.io/projected/6db8e691-22a1-48fc-9edc-2454692a72f0-kube-api-access-wr2md\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.747394 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.747373 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.747470 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.747439 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.747508 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.747499 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.747566 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.747543 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.747722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.747583 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db8e691-22a1-48fc-9edc-2454692a72f0-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.747722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.747625 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hztsv\" (UniqueName: \"kubernetes.io/projected/f30d7145-f5ba-4051-ac0b-371c2a6f2334-kube-api-access-hztsv\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.747722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.747674 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.747908 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.747846 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.747908 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.747893 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.748019 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.747923 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.748019 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.747959 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.848764 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.848692 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.848764 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.848767 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.849024 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.848824 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr2md\" (UniqueName: \"kubernetes.io/projected/6db8e691-22a1-48fc-9edc-2454692a72f0-kube-api-access-wr2md\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.849024 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.848850 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.849024 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.848886 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.849024 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.848972 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.849024 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.849015 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.849284 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.849043 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db8e691-22a1-48fc-9edc-2454692a72f0-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.849284 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.849085 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hztsv\" (UniqueName: \"kubernetes.io/projected/f30d7145-f5ba-4051-ac0b-371c2a6f2334-kube-api-access-hztsv\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.849284 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.849134 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.849284 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.849167 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.849284 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.849207 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.849284 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.849236 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.849284 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.849273 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.849683 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.849333 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.849683 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.849391 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.849683 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.849425 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.849683 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.849503 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.850200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.850168 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.850314 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.850219 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.851427 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.850681 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-workload-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.851427 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.850848 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.851427 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.850960 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db8e691-22a1-48fc-9edc-2454692a72f0-istiod-ca-cert\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.851427 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.851004 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.851427 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.851024 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.851427 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.851334 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-credential-socket\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.851427 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.851376 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-workload-certs\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.851427 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.851386 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-data\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.852486 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.852460 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.852486 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.852471 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-envoy\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.852660 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.852644 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.852826 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.852807 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-podinfo\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.874476 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.874449 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.874651 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.874635 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-token\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.875254 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.875237 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr2md\" (UniqueName: \"kubernetes.io/projected/6db8e691-22a1-48fc-9edc-2454692a72f0-kube-api-access-wr2md\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.875725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.875676 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hztsv\" (UniqueName: \"kubernetes.io/projected/f30d7145-f5ba-4051-ac0b-371c2a6f2334-kube-api-access-hztsv\") pod \"e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:29.962505 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.962418 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:29.978501 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:29.978471 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:30.051861 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:30.051680 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 16:40:30.051861 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:30.051829 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" podUID="38262c13-420b-43af-bb5e-1cb3c62e44b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 16:40:30.159375 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:40:30.159298 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:40:30.328685 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:30.328642 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw"] Nov 03 16:40:30.330105 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:40:30.330077 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db8e691_22a1_48fc_9edc_2454692a72f0.slice/crio-a114746012a81f93150ee2f0b9b8660de266f3ee6930589876487a7cc6f76943 WatchSource:0}: Error finding container a114746012a81f93150ee2f0b9b8660de266f3ee6930589876487a7cc6f76943: Status 404 returned error can't find the container with id a114746012a81f93150ee2f0b9b8660de266f3ee6930589876487a7cc6f76943 Nov 03 16:40:30.333175 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:30.333135 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:40:30.333260 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:30.333220 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:40:30.333309 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:30.333263 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:40:30.358943 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:30.358919 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf"] Nov 03 16:40:30.361986 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:40:30.361962 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30d7145_f5ba_4051_ac0b_371c2a6f2334.slice/crio-41dfb53fffee41e451f847770640bfcaed995b573f036a69757446931d8c3679 WatchSource:0}: Error finding container 41dfb53fffee41e451f847770640bfcaed995b573f036a69757446931d8c3679: Status 404 returned error can't find the container with id 41dfb53fffee41e451f847770640bfcaed995b573f036a69757446931d8c3679 Nov 03 16:40:30.364742 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:30.364675 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:40:30.364848 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:30.364791 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:40:30.364848 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:30.364832 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:40:31.052070 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:31.052024 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 16:40:31.052378 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:31.052339 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" podUID="38262c13-420b-43af-bb5e-1cb3c62e44b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 16:40:31.196780 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:31.196744 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" event={"ID":"6db8e691-22a1-48fc-9edc-2454692a72f0","Type":"ContainerStarted","Data":"f2cac98e47b7c2f8fc1649a41dcb68cf124bff1b04a0321a990068e3eafef93e"} Nov 03 16:40:31.197165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:31.196810 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" event={"ID":"6db8e691-22a1-48fc-9edc-2454692a72f0","Type":"ContainerStarted","Data":"a114746012a81f93150ee2f0b9b8660de266f3ee6930589876487a7cc6f76943"} Nov 03 16:40:31.200765 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:31.198654 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" event={"ID":"f30d7145-f5ba-4051-ac0b-371c2a6f2334","Type":"ContainerStarted","Data":"2288c8892517dad71363aa8c75bdd838efa698174567b6ef6e6b39f1929477dc"} Nov 03 16:40:31.200765 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:31.198752 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" event={"ID":"f30d7145-f5ba-4051-ac0b-371c2a6f2334","Type":"ContainerStarted","Data":"41dfb53fffee41e451f847770640bfcaed995b573f036a69757446931d8c3679"} Nov 03 16:40:31.262115 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:31.262057 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" podStartSLOduration=2.262042304 podStartE2EDuration="2.262042304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:31.261282264 +0000 UTC m=+1574.850282237" watchObservedRunningTime="2026-11-03 16:40:31.262042304 +0000 UTC m=+1574.851042273" Nov 03 16:40:31.963792 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:31.962953 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:31.967923 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:31.967884 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:40:31.968148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:31.967953 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" podUID="6db8e691-22a1-48fc-9edc-2454692a72f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 16:40:31.978788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:31.978752 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:31.982132 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:31.982093 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:40:31.982592 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:31.982470 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" podUID="f30d7145-f5ba-4051-ac0b-371c2a6f2334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 16:40:32.382918 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:32.382887 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:32.384828 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:32.384690 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:32.386095 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:32.386078 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:40:32.465941 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:32.465863 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" podStartSLOduration=3.465845831 podStartE2EDuration="3.465845831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:31.329777469 +0000 UTC m=+1574.918777439" watchObservedRunningTime="2026-11-03 16:40:32.465845831 +0000 UTC m=+1576.054845800" Nov 03 16:40:32.994786 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:32.993630 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:40:32.994786 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:32.993746 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" podUID="6db8e691-22a1-48fc-9edc-2454692a72f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 16:40:32.996495 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:32.996033 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:40:32.996869 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:32.996806 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" podUID="f30d7145-f5ba-4051-ac0b-371c2a6f2334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 16:40:33.963143 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:33.963103 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:40:33.963581 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:33.963167 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" podUID="6db8e691-22a1-48fc-9edc-2454692a72f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 16:40:33.978978 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:33.978839 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:40:33.978978 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:33.978909 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" podUID="f30d7145-f5ba-4051-ac0b-371c2a6f2334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 16:40:34.967677 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:34.966841 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:40:34.967677 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:34.966900 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" podUID="6db8e691-22a1-48fc-9edc-2454692a72f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 16:40:34.980653 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:34.980615 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:40:34.980952 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:34.980924 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" podUID="f30d7145-f5ba-4051-ac0b-371c2a6f2334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 16:40:35.963034 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:35.962988 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:40:35.963243 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:35.963055 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" podUID="6db8e691-22a1-48fc-9edc-2454692a72f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 16:40:35.978792 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:35.978761 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:40:35.979191 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:35.978814 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" podUID="f30d7145-f5ba-4051-ac0b-371c2a6f2334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 16:40:36.963901 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:36.963860 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 16:40:36.964121 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:36.963956 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" podUID="6db8e691-22a1-48fc-9edc-2454692a72f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 16:40:36.978931 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:36.978870 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 16:40:36.979448 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:36.978961 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" podUID="f30d7145-f5ba-4051-ac0b-371c2a6f2334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 16:40:38.260629 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:38.260597 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:38.262098 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:38.262074 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:38.264034 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:38.264011 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 16:40:38.359730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:38.359656 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:38.360819 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:38.360793 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:38.362041 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:38.362018 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:40:41.991832 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:41.991797 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:40:45.274730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.273742 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx"] Nov 03 16:40:45.372796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.304334 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx" Nov 03 16:40:45.372796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.371667 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8650a6e8-dockercfg-6bg8h\"" Nov 03 16:40:45.372796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.372277 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8650a6e8\"" Nov 03 16:40:45.380746 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.378963 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b"] Nov 03 16:40:45.407674 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.407283 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" Nov 03 16:40:45.408901 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.408871 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx"] Nov 03 16:40:45.775382 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.693283 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9158c889-3243-4483-a481-3b41f97f0350-plugins-config-volume\") pod \"payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b\" (UID: \"9158c889-3243-4483-a481-3b41f97f0350\") " pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" Nov 03 16:40:45.775382 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.693335 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvctf\" (UniqueName: \"kubernetes.io/projected/5af158d1-c8f3-42ff-89f6-cac3af71192b-kube-api-access-rvctf\") pod \"payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx\" (UID: \"5af158d1-c8f3-42ff-89f6-cac3af71192b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx" Nov 03 16:40:45.775382 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.693424 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztj4v\" (UniqueName: \"kubernetes.io/projected/9158c889-3243-4483-a481-3b41f97f0350-kube-api-access-ztj4v\") pod \"payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b\" (UID: \"9158c889-3243-4483-a481-3b41f97f0350\") " pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" Nov 03 16:40:45.775382 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.693476 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af158d1-c8f3-42ff-89f6-cac3af71192b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx\" (UID: \"5af158d1-c8f3-42ff-89f6-cac3af71192b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx" Nov 03 16:40:45.775382 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.724945 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b"] Nov 03 16:40:45.815927 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.815886 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9158c889-3243-4483-a481-3b41f97f0350-plugins-config-volume\") pod \"payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b\" (UID: \"9158c889-3243-4483-a481-3b41f97f0350\") " pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" Nov 03 16:40:45.820144 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.817342 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9158c889-3243-4483-a481-3b41f97f0350-plugins-config-volume\") pod \"payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b\" (UID: \"9158c889-3243-4483-a481-3b41f97f0350\") " pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" Nov 03 16:40:45.821023 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.820993 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvctf\" (UniqueName: \"kubernetes.io/projected/5af158d1-c8f3-42ff-89f6-cac3af71192b-kube-api-access-rvctf\") pod \"payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx\" (UID: \"5af158d1-c8f3-42ff-89f6-cac3af71192b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx" Nov 03 16:40:45.821384 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.821359 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztj4v\" (UniqueName: \"kubernetes.io/projected/9158c889-3243-4483-a481-3b41f97f0350-kube-api-access-ztj4v\") pod \"payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b\" (UID: \"9158c889-3243-4483-a481-3b41f97f0350\") " pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" Nov 03 16:40:45.822254 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.822232 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af158d1-c8f3-42ff-89f6-cac3af71192b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx\" (UID: \"5af158d1-c8f3-42ff-89f6-cac3af71192b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx" Nov 03 16:40:45.823689 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.823322 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af158d1-c8f3-42ff-89f6-cac3af71192b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx\" (UID: \"5af158d1-c8f3-42ff-89f6-cac3af71192b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx" Nov 03 16:40:45.847863 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.847659 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvctf\" (UniqueName: \"kubernetes.io/projected/5af158d1-c8f3-42ff-89f6-cac3af71192b-kube-api-access-rvctf\") pod \"payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx\" (UID: \"5af158d1-c8f3-42ff-89f6-cac3af71192b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx" Nov 03 16:40:45.849311 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:45.848594 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztj4v\" (UniqueName: \"kubernetes.io/projected/9158c889-3243-4483-a481-3b41f97f0350-kube-api-access-ztj4v\") pod \"payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b\" (UID: \"9158c889-3243-4483-a481-3b41f97f0350\") " pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" Nov 03 16:40:46.079832 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:46.079750 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx" Nov 03 16:40:46.098210 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:46.098179 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" Nov 03 16:40:46.401093 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:46.401060 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx"] Nov 03 16:40:46.401411 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:40:46.401269 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af158d1_c8f3_42ff_89f6_cac3af71192b.slice/crio-47145e27f31c250138404fee0f776e52aba1afea4084d990b3b2ca3230233fdf WatchSource:0}: Error finding container 47145e27f31c250138404fee0f776e52aba1afea4084d990b3b2ca3230233fdf: Status 404 returned error can't find the container with id 47145e27f31c250138404fee0f776e52aba1afea4084d990b3b2ca3230233fdf Nov 03 16:40:46.446517 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:40:46.446488 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9158c889_3243_4483_a481_3b41f97f0350.slice/crio-4701198e237f67e296b16dd178f9ca1b3e5cbdeb33f14d4a366270ec65435e03 WatchSource:0}: Error finding container 4701198e237f67e296b16dd178f9ca1b3e5cbdeb33f14d4a366270ec65435e03: Status 404 returned error can't find the container with id 4701198e237f67e296b16dd178f9ca1b3e5cbdeb33f14d4a366270ec65435e03 Nov 03 16:40:46.448352 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:46.448249 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b"] Nov 03 16:40:46.784874 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:46.784838 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx" event={"ID":"5af158d1-c8f3-42ff-89f6-cac3af71192b","Type":"ContainerStarted","Data":"2bd0f09e102e27fa99c1d9a05cc286e47645264123e55e17c5469b7759c3bbb6"} Nov 03 16:40:46.785077 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:46.784896 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx" event={"ID":"5af158d1-c8f3-42ff-89f6-cac3af71192b","Type":"ContainerStarted","Data":"47145e27f31c250138404fee0f776e52aba1afea4084d990b3b2ca3230233fdf"} Nov 03 16:40:46.785077 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:46.784937 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx" Nov 03 16:40:46.786576 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:46.786556 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" event={"ID":"9158c889-3243-4483-a481-3b41f97f0350","Type":"ContainerStarted","Data":"471d258dfb88ae032f47b3672f55100023e37d58a3abefa08c02587a0cb72f5d"} Nov 03 16:40:46.786627 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:46.786595 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" event={"ID":"9158c889-3243-4483-a481-3b41f97f0350","Type":"ContainerStarted","Data":"4701198e237f67e296b16dd178f9ca1b3e5cbdeb33f14d4a366270ec65435e03"} Nov 03 16:40:46.786627 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:46.786616 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" Nov 03 16:40:46.852115 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:46.852059 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx" podStartSLOduration=1.852043036 podStartE2EDuration="1.852043036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:46.850784422 +0000 UTC m=+1590.439784392" watchObservedRunningTime="2026-11-03 16:40:46.852043036 +0000 UTC m=+1590.441043004" Nov 03 16:40:46.906351 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:46.906288 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" podStartSLOduration=1.9062716640000001 podStartE2EDuration="1.906271664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:46.904818203 +0000 UTC m=+1590.493818172" watchObservedRunningTime="2026-11-03 16:40:46.906271664 +0000 UTC m=+1590.495271632" Nov 03 16:40:49.736549 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.736512 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4"] Nov 03 16:40:49.741854 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.741827 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.759690 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.759661 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9501ab-openshift-default-dockercfg-vmf49\"" Nov 03 16:40:49.778236 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.778202 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4"] Nov 03 16:40:49.821282 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.821251 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7klr\" (UniqueName: \"kubernetes.io/projected/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-kube-api-access-s7klr\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.821282 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.821286 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-credential-socket\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.821510 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.821417 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-envoy\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.821510 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.821441 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-data\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.821622 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.821592 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-workload-certs\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.821765 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.821748 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istiod-ca-cert\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.821842 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.821789 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-podinfo\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.821922 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.821909 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-workload-socket\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.821958 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.821930 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-token\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.941721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.934618 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-workload-socket\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.941721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.934679 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-token\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.941721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.934786 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7klr\" (UniqueName: \"kubernetes.io/projected/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-kube-api-access-s7klr\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.941721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.934815 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-credential-socket\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.941721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.934916 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-envoy\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.941721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.934944 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-data\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.941721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.935055 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-workload-certs\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.941721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.935149 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istiod-ca-cert\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.941721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.935188 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-podinfo\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.953723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.947819 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-workload-socket\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.953723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.948138 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-workload-certs\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.953723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.948390 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-data\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.953723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.949882 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-credential-socket\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:49.959592 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:49.958322 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istiod-ca-cert\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:50.097476 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:50.097442 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7klr\" (UniqueName: \"kubernetes.io/projected/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-kube-api-access-s7klr\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:50.106407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:50.106372 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-podinfo\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:50.110380 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:50.110345 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-envoy\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:50.110719 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:50.110518 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-token\") pod \"e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:50.361275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:50.358325 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:50.696848 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:40:50.696821 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd552ecdb_c3a5_4ee8_ad18_fe6809cbf03c.slice/crio-9165c20a4115f4400a60dc6e1c4cec1bfd0ff40c89ba92c78c2b686016841cfc WatchSource:0}: Error finding container 9165c20a4115f4400a60dc6e1c4cec1bfd0ff40c89ba92c78c2b686016841cfc: Status 404 returned error can't find the container with id 9165c20a4115f4400a60dc6e1c4cec1bfd0ff40c89ba92c78c2b686016841cfc Nov 03 16:40:50.697241 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:50.697219 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4"] Nov 03 16:40:50.699330 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:50.699298 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:40:50.699394 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:50.699363 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:40:50.699447 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:50.699413 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:40:50.819466 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:50.819425 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" event={"ID":"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c","Type":"ContainerStarted","Data":"bc2c45d2210c6b7acad5cfde81d9657646c80437105523dad5fe21285aba3d44"} Nov 03 16:40:50.819963 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:50.819493 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" event={"ID":"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c","Type":"ContainerStarted","Data":"9165c20a4115f4400a60dc6e1c4cec1bfd0ff40c89ba92c78c2b686016841cfc"} Nov 03 16:40:50.872231 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:50.872175 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" podStartSLOduration=1.872161303 podStartE2EDuration="1.872161303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:40:50.869236721 +0000 UTC m=+1594.458236691" watchObservedRunningTime="2026-11-03 16:40:50.872161303 +0000 UTC m=+1594.461161273" Nov 03 16:40:51.358897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:51.358860 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:51.365128 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:51.365101 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:51.825441 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:51.825407 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:51.826630 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:51.826608 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:40:57.800432 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:57.800399 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" Nov 03 16:40:57.800942 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:40:57.800473 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx" Nov 03 16:41:00.134759 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:41:00.134721 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:41:04.128000 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:04.126798 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:05.177849 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:05.177780 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:05.183728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:05.178765 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:41:05.268499 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:05.266062 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:41:05.615674 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:05.615060 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:41:16.924485 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:16.924443 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:41:20.635345 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.635310 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2"] Nov 03 16:41:20.642175 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.642152 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.682730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.682668 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8650a6e8\"/\"e2e-lifecycle-model-8650a6e8-kserve-self-signed-certs\"" Nov 03 16:41:20.682730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.682717 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8650a6e8\"/\"kube-root-ca.crt\"" Nov 03 16:41:20.682945 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.682663 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8650a6e8\"/\"openshift-service-ca.crt\"" Nov 03 16:41:20.683066 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.683049 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8650a6e8\"/\"default-dockercfg-z9v98\"" Nov 03 16:41:20.686013 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.685988 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2"] Nov 03 16:41:20.763879 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.763845 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-tmp-dir\") pod \"e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.763879 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.763883 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-home\") pod \"e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.764157 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.763960 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-dshm\") pod \"e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.764157 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.764054 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-tls-certs\") pod \"e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.764157 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.764106 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-model-cache\") pod \"e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.764256 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.764232 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gstk2\" (UniqueName: \"kubernetes.io/projected/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-kube-api-access-gstk2\") pod \"e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.865130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.865096 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-dshm\") pod \"e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.865329 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.865157 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-tls-certs\") pod \"e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.865329 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.865198 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-model-cache\") pod \"e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.865443 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.865345 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gstk2\" (UniqueName: \"kubernetes.io/projected/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-kube-api-access-gstk2\") pod \"e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.865562 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.865477 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-tmp-dir\") pod \"e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.865562 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.865506 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-home\") pod \"e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.865669 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.865616 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-model-cache\") pod \"e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.865875 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.865851 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-tmp-dir\") pod \"e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.865985 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.865876 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-home\") pod \"e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.867786 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.867763 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-tls-certs\") pod \"e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.867887 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.867868 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-dshm\") pod \"e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.954778 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.944492 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gstk2\" (UniqueName: \"kubernetes.io/projected/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-kube-api-access-gstk2\") pod \"e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:20.978451 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:20.977047 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:21.566577 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:21.566350 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2"] Nov 03 16:41:21.573579 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:21.573547 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" event={"ID":"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8","Type":"ContainerStarted","Data":"334dcca0a1a1008a2d89b7407ab1467de93f88a4b8c47b663fbe323616e41c77"} Nov 03 16:41:22.581355 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:22.581309 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" event={"ID":"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8","Type":"ContainerStarted","Data":"8f9ba78da88a969087d405bbb6467967a83f50193a8652b3f920110c3ac024ef"} Nov 03 16:41:22.629323 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:22.629273 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" podStartSLOduration=2.62925838 podStartE2EDuration="2.62925838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:41:22.62590375 +0000 UTC m=+1626.214903720" watchObservedRunningTime="2026-11-03 16:41:22.62925838 +0000 UTC m=+1626.218258347" Nov 03 16:41:22.979193 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:22.979051 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:23.967179 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:23.967142 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb"] Nov 03 16:41:23.971415 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:23.971395 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:23.996914 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:23.996889 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8650a6e8-openshift-default-dockercfg-dq7gn\"" Nov 03 16:41:24.013565 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.013537 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb"] Nov 03 16:41:24.027882 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.027849 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3de2da9-631c-4ff3-96d3-8cf575990923-istiod-ca-cert\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.028215 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.028190 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-workload-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.028636 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.028590 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-envoy\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.028954 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.028921 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-token\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.029580 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.029560 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-data\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.029902 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.029881 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-workload-certs\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.030420 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.030401 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-podinfo\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.030588 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.030440 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jblr2\" (UniqueName: \"kubernetes.io/projected/f3de2da9-631c-4ff3-96d3-8cf575990923-kube-api-access-jblr2\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.030588 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.030484 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-credential-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.131770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.131734 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-envoy\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.131957 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.131799 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-token\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.131957 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.131879 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-data\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.131957 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.131906 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-workload-certs\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.131957 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.131939 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-podinfo\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.132198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.131971 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jblr2\" (UniqueName: \"kubernetes.io/projected/f3de2da9-631c-4ff3-96d3-8cf575990923-kube-api-access-jblr2\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.132198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.132019 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-credential-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.132198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.132139 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3de2da9-631c-4ff3-96d3-8cf575990923-istiod-ca-cert\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.132401 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.132300 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-workload-certs\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.132401 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.132366 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-data\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.132516 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.132435 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-workload-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.132601 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.132572 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-credential-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.132771 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.132748 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-workload-socket\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.133397 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.133374 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3de2da9-631c-4ff3-96d3-8cf575990923-istiod-ca-cert\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.134975 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.134951 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-envoy\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.135557 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.135531 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-podinfo\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.159810 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.159777 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jblr2\" (UniqueName: \"kubernetes.io/projected/f3de2da9-631c-4ff3-96d3-8cf575990923-kube-api-access-jblr2\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.161174 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.161152 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-token\") pod \"e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.287885 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.287856 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:24.613239 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:41:24.613212 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3de2da9_631c_4ff3_96d3_8cf575990923.slice/crio-2be5b8aab8f838ec302765a56765d2172d7eabf80d2592da448fcc5638cbd899 WatchSource:0}: Error finding container 2be5b8aab8f838ec302765a56765d2172d7eabf80d2592da448fcc5638cbd899: Status 404 returned error can't find the container with id 2be5b8aab8f838ec302765a56765d2172d7eabf80d2592da448fcc5638cbd899 Nov 03 16:41:24.616023 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.615998 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb"] Nov 03 16:41:24.616386 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.616348 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:41:24.616456 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.616433 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:41:24.616494 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:24.616478 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:41:25.605186 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:25.605148 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" event={"ID":"f3de2da9-631c-4ff3-96d3-8cf575990923","Type":"ContainerStarted","Data":"be798cf05b2c408523f3672273d454f1b1a6726878a74e10e7d9575bb687d4f9"} Nov 03 16:41:25.605613 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:25.605206 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" event={"ID":"f3de2da9-631c-4ff3-96d3-8cf575990923","Type":"ContainerStarted","Data":"2be5b8aab8f838ec302765a56765d2172d7eabf80d2592da448fcc5638cbd899"} Nov 03 16:41:25.667426 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:25.667367 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" podStartSLOduration=2.667350794 podStartE2EDuration="2.667350794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:41:25.666676671 +0000 UTC m=+1629.255676638" watchObservedRunningTime="2026-11-03 16:41:25.667350794 +0000 UTC m=+1629.256350760" Nov 03 16:41:26.288539 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:26.288504 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:26.289954 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:26.289929 4947 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 16:41:26.290095 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:26.289990 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" podUID="f3de2da9-631c-4ff3-96d3-8cf575990923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 16:41:27.288184 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:27.288154 4947 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 16:41:27.288588 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:27.288204 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" podUID="f3de2da9-631c-4ff3-96d3-8cf575990923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 16:41:28.289100 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:28.289060 4947 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 16:41:28.289650 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:28.289116 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" podUID="f3de2da9-631c-4ff3-96d3-8cf575990923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 16:41:28.643562 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:28.643523 4947 generic.go:356] "Generic (PLEG): container finished" podID="d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" containerID="bc2c45d2210c6b7acad5cfde81d9657646c80437105523dad5fe21285aba3d44" exitCode=0 Nov 03 16:41:28.643759 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:28.643590 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" event={"ID":"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c","Type":"ContainerDied","Data":"bc2c45d2210c6b7acad5cfde81d9657646c80437105523dad5fe21285aba3d44"} Nov 03 16:41:28.644012 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:28.643999 4947 scope.go:122] "RemoveContainer" containerID="bc2c45d2210c6b7acad5cfde81d9657646c80437105523dad5fe21285aba3d44" Nov 03 16:41:28.645250 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:28.645224 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:41:28.645302 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:28.645288 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:41:28.645335 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:28.645319 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:41:29.288397 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.288365 4947 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 16:41:29.288585 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.288430 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" podUID="f3de2da9-631c-4ff3-96d3-8cf575990923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 16:41:29.660724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.660612 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" event={"ID":"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c","Type":"ContainerStarted","Data":"4254b24b4d387b8b283574032703df31f1cf632a9b7c78da9718024bc2f4dc08"} Nov 03 16:41:29.852555 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.852518 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp"] Nov 03 16:41:29.858876 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.858810 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.864105 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.864076 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-token\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.864105 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.864110 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-podinfo\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.864350 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.864133 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-workload-socket\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.864350 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.864179 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istiod-ca-cert\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.864350 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.864203 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-credential-socket\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.864350 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.864220 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-data\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.864350 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.864246 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-workload-certs\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.864619 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.864359 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-envoy\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.864619 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.864426 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-596ft\" (UniqueName: \"kubernetes.io/projected/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-kube-api-access-596ft\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.876114 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.876084 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d89163-openshift-default-dockercfg-vsktx\"" Nov 03 16:41:29.892383 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.892343 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp"] Nov 03 16:41:29.981853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.978793 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-envoy\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.981853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.978841 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596ft\" (UniqueName: \"kubernetes.io/projected/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-kube-api-access-596ft\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.981853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.978899 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-token\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.981853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.978928 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-podinfo\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.981853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.978966 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-workload-socket\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.981853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.979056 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istiod-ca-cert\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.981853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.979097 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-credential-socket\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.981853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.979122 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-data\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.981853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.979156 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-workload-certs\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:29.981853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:29.979862 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-workload-certs\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:30.055719 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.053111 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-workload-socket\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:30.101737 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.101684 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istiod-ca-cert\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:30.102291 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.102265 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-credential-socket\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:30.102681 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.102662 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-data\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:30.103473 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.103452 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-envoy\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:30.140757 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.140716 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-token\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:30.150063 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.150030 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-podinfo\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:30.180721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.177899 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-596ft\" (UniqueName: \"kubernetes.io/projected/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-kube-api-access-596ft\") pod \"e2e-api-b-d89163-openshift-default-69879764b9-h8szp\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:30.180721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.178010 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:30.312723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.312031 4947 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 16:41:30.312723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.312092 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" podUID="f3de2da9-631c-4ff3-96d3-8cf575990923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 16:41:30.360451 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.360216 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:41:30.360451 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.360331 4947 patch_prober.go:29] interesting pod/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 16:41:30.360451 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.360387 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" podUID="d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 16:41:30.416724 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:41:30.406002 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:41:30.952966 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.952679 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp"] Nov 03 16:41:30.954526 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:41:30.954500 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2aa7e1_3ff0_469b_9cb1_418031bf90ef.slice/crio-8a6fe6ed4f3d6ff1aa2a1ca7103ee364400aa36593c1bbd333859b0ffe79ebb1 WatchSource:0}: Error finding container 8a6fe6ed4f3d6ff1aa2a1ca7103ee364400aa36593c1bbd333859b0ffe79ebb1: Status 404 returned error can't find the container with id 8a6fe6ed4f3d6ff1aa2a1ca7103ee364400aa36593c1bbd333859b0ffe79ebb1 Nov 03 16:41:30.957058 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.957023 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:41:30.957126 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.957093 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:41:30.957126 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.957121 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:41:30.978778 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.978753 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:30.992357 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:30.992323 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:31.301723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:31.290142 4947 patch_prober.go:29] interesting pod/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Nov 03 16:41:31.301723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:31.290206 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" podUID="f3de2da9-631c-4ff3-96d3-8cf575990923" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Nov 03 16:41:31.372723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:31.359417 4947 patch_prober.go:29] interesting pod/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 16:41:31.372723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:31.359474 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" podUID="d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 16:41:31.731780 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:31.724992 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" event={"ID":"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef","Type":"ContainerStarted","Data":"cf985113eb07ee321b5f96576d58fe60cd3b6d258ff883be46a11242203c92ef"} Nov 03 16:41:31.731780 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:31.725078 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" event={"ID":"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef","Type":"ContainerStarted","Data":"8a6fe6ed4f3d6ff1aa2a1ca7103ee364400aa36593c1bbd333859b0ffe79ebb1"} Nov 03 16:41:31.749597 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:31.749564 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:41:31.776277 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:31.776217 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" podStartSLOduration=2.7762016689999998 podStartE2EDuration="2.776201669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:41:31.773438192 +0000 UTC m=+1635.362438159" watchObservedRunningTime="2026-11-03 16:41:31.776201669 +0000 UTC m=+1635.365201700" Nov 03 16:41:32.181187 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:32.181158 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:32.186925 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:32.186901 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:32.293511 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:32.293486 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:32.293932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:32.293915 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:32.294663 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:32.294643 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:41:32.359799 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:32.359770 4947 patch_prober.go:29] interesting pod/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 16:41:32.359960 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:32.359820 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" podUID="d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 16:41:32.729212 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:32.729174 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:32.730374 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:32.730350 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:41:33.368224 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:33.368189 4947 patch_prober.go:29] interesting pod/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 16:41:33.369148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:33.369090 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" podUID="d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 16:41:34.359387 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:34.359355 4947 patch_prober.go:29] interesting pod/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Nov 03 16:41:34.359595 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:34.359417 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" podUID="d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Nov 03 16:41:35.374687 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:35.374654 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:41:35.375971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:35.375447 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:41:35.376664 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:35.376641 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:41:36.399733 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.397528 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps"] Nov 03 16:41:36.409141 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.409101 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps" Nov 03 16:41:36.428255 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.419040 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n"] Nov 03 16:41:36.448416 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.448371 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d89163-dockercfg-d8dcj\"" Nov 03 16:41:36.448684 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.448664 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d89163\"" Nov 03 16:41:36.488466 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.487970 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps"] Nov 03 16:41:36.488466 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.488015 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n"] Nov 03 16:41:36.488466 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.488078 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n" Nov 03 16:41:36.509915 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.509882 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d560fb8-8a25-43a4-9327-0e59687c3bd9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps\" (UID: \"5d560fb8-8a25-43a4-9327-0e59687c3bd9\") " pod="openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps" Nov 03 16:41:36.510169 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.510151 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkv28\" (UniqueName: \"kubernetes.io/projected/5d560fb8-8a25-43a4-9327-0e59687c3bd9-kube-api-access-kkv28\") pod \"payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps\" (UID: \"5d560fb8-8a25-43a4-9327-0e59687c3bd9\") " pod="openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps" Nov 03 16:41:36.717662 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.717559 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58e71f9-4452-4e6a-a70d-642ab5815865-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n\" (UID: \"f58e71f9-4452-4e6a-a70d-642ab5815865\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n" Nov 03 16:41:36.752921 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.752883 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d560fb8-8a25-43a4-9327-0e59687c3bd9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps\" (UID: \"5d560fb8-8a25-43a4-9327-0e59687c3bd9\") " pod="openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps" Nov 03 16:41:36.753254 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.753178 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkv28\" (UniqueName: \"kubernetes.io/projected/5d560fb8-8a25-43a4-9327-0e59687c3bd9-kube-api-access-kkv28\") pod \"payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps\" (UID: \"5d560fb8-8a25-43a4-9327-0e59687c3bd9\") " pod="openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps" Nov 03 16:41:36.753436 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.753415 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z68x\" (UniqueName: \"kubernetes.io/projected/f58e71f9-4452-4e6a-a70d-642ab5815865-kube-api-access-5z68x\") pod \"payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n\" (UID: \"f58e71f9-4452-4e6a-a70d-642ab5815865\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n" Nov 03 16:41:36.804580 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.804192 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d560fb8-8a25-43a4-9327-0e59687c3bd9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps\" (UID: \"5d560fb8-8a25-43a4-9327-0e59687c3bd9\") " pod="openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps" Nov 03 16:41:36.808639 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.808605 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkv28\" (UniqueName: \"kubernetes.io/projected/5d560fb8-8a25-43a4-9327-0e59687c3bd9-kube-api-access-kkv28\") pod \"payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps\" (UID: \"5d560fb8-8a25-43a4-9327-0e59687c3bd9\") " pod="openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps" Nov 03 16:41:36.856229 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.856187 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58e71f9-4452-4e6a-a70d-642ab5815865-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n\" (UID: \"f58e71f9-4452-4e6a-a70d-642ab5815865\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n" Nov 03 16:41:36.856439 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.856387 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z68x\" (UniqueName: \"kubernetes.io/projected/f58e71f9-4452-4e6a-a70d-642ab5815865-kube-api-access-5z68x\") pod \"payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n\" (UID: \"f58e71f9-4452-4e6a-a70d-642ab5815865\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n" Nov 03 16:41:36.857091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.857065 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58e71f9-4452-4e6a-a70d-642ab5815865-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n\" (UID: \"f58e71f9-4452-4e6a-a70d-642ab5815865\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n" Nov 03 16:41:36.881740 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:36.881642 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z68x\" (UniqueName: \"kubernetes.io/projected/f58e71f9-4452-4e6a-a70d-642ab5815865-kube-api-access-5z68x\") pod \"payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n\" (UID: \"f58e71f9-4452-4e6a-a70d-642ab5815865\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n" Nov 03 16:41:37.053214 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:37.053179 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps" Nov 03 16:41:37.108267 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:37.108231 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n" Nov 03 16:41:37.410724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:37.402159 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps"] Nov 03 16:41:37.462721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:37.462683 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n"] Nov 03 16:41:37.470947 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:41:37.470918 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58e71f9_4452_4e6a_a70d_642ab5815865.slice/crio-821d48b1bc812d027ae612aadebd43aaee324a3e44c8970d8b7c0b9d6a42c3b0 WatchSource:0}: Error finding container 821d48b1bc812d027ae612aadebd43aaee324a3e44c8970d8b7c0b9d6a42c3b0: Status 404 returned error can't find the container with id 821d48b1bc812d027ae612aadebd43aaee324a3e44c8970d8b7c0b9d6a42c3b0 Nov 03 16:41:37.814762 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:37.814684 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n" event={"ID":"f58e71f9-4452-4e6a-a70d-642ab5815865","Type":"ContainerStarted","Data":"df5bac6bc2c019ee6917e12f8acef5b2c454ec1f27367f8e884c3da9375c41aa"} Nov 03 16:41:37.814953 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:37.814789 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n" Nov 03 16:41:37.814953 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:37.814817 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n" event={"ID":"f58e71f9-4452-4e6a-a70d-642ab5815865","Type":"ContainerStarted","Data":"821d48b1bc812d027ae612aadebd43aaee324a3e44c8970d8b7c0b9d6a42c3b0"} Nov 03 16:41:37.816792 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:37.816756 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps" event={"ID":"5d560fb8-8a25-43a4-9327-0e59687c3bd9","Type":"ContainerStarted","Data":"1a8925df267cd9a298fe5870275dece9de1546e03ff8265b981068bb2f917df2"} Nov 03 16:41:37.816907 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:37.816810 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps" event={"ID":"5d560fb8-8a25-43a4-9327-0e59687c3bd9","Type":"ContainerStarted","Data":"0d1f10b2e444d9a662323873cff6de2d5433bea8e62a4fedecaf8d59ce74f164"} Nov 03 16:41:37.816907 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:37.816838 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps" Nov 03 16:41:37.862602 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:37.862544 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n" podStartSLOduration=1.862530852 podStartE2EDuration="1.862530852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:41:37.861409278 +0000 UTC m=+1641.450409248" watchObservedRunningTime="2026-11-03 16:41:37.862530852 +0000 UTC m=+1641.451530819" Nov 03 16:41:37.917797 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:37.917735 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps" podStartSLOduration=1.917718853 podStartE2EDuration="1.917718853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:41:37.915891557 +0000 UTC m=+1641.504891524" watchObservedRunningTime="2026-11-03 16:41:37.917718853 +0000 UTC m=+1641.506718823" Nov 03 16:41:42.024918 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:42.024885 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:41:48.825987 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:48.825954 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps" Nov 03 16:41:48.826368 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:41:48.826031 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n" Nov 03 16:42:00.130526 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:42:00.130447 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:42:25.427154 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.427116 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6"] Nov 03 16:42:25.433444 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.433414 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6" Nov 03 16:42:25.454799 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.454761 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6"] Nov 03 16:42:25.464514 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.464467 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-6fa98e\"" Nov 03 16:42:25.468502 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.468473 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d"] Nov 03 16:42:25.473349 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.473324 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d" Nov 03 16:42:25.497407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.497375 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d"] Nov 03 16:42:25.499765 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.499739 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-6fa98e-dockercfg-xrbwx\"" Nov 03 16:42:25.582720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.582045 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0781fe8-13dc-4f23-9574-9967398b0a91-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6\" (UID: \"f0781fe8-13dc-4f23-9574-9967398b0a91\") " pod="openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6" Nov 03 16:42:25.582720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.582242 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6cmh\" (UniqueName: \"kubernetes.io/projected/6560e01c-5031-49da-aea1-19d2df514d44-kube-api-access-d6cmh\") pod \"payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d\" (UID: \"6560e01c-5031-49da-aea1-19d2df514d44\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d" Nov 03 16:42:25.582720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.582305 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsw4p\" (UniqueName: \"kubernetes.io/projected/f0781fe8-13dc-4f23-9574-9967398b0a91-kube-api-access-bsw4p\") pod \"payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6\" (UID: \"f0781fe8-13dc-4f23-9574-9967398b0a91\") " pod="openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6" Nov 03 16:42:25.582720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.582458 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6560e01c-5031-49da-aea1-19d2df514d44-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d\" (UID: \"6560e01c-5031-49da-aea1-19d2df514d44\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d" Nov 03 16:42:25.686766 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.684173 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6cmh\" (UniqueName: \"kubernetes.io/projected/6560e01c-5031-49da-aea1-19d2df514d44-kube-api-access-d6cmh\") pod \"payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d\" (UID: \"6560e01c-5031-49da-aea1-19d2df514d44\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d" Nov 03 16:42:25.686766 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.684253 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsw4p\" (UniqueName: \"kubernetes.io/projected/f0781fe8-13dc-4f23-9574-9967398b0a91-kube-api-access-bsw4p\") pod \"payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6\" (UID: \"f0781fe8-13dc-4f23-9574-9967398b0a91\") " pod="openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6" Nov 03 16:42:25.686766 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.684399 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6560e01c-5031-49da-aea1-19d2df514d44-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d\" (UID: \"6560e01c-5031-49da-aea1-19d2df514d44\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d" Nov 03 16:42:25.686766 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.684476 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0781fe8-13dc-4f23-9574-9967398b0a91-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6\" (UID: \"f0781fe8-13dc-4f23-9574-9967398b0a91\") " pod="openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6" Nov 03 16:42:25.686766 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.685548 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0781fe8-13dc-4f23-9574-9967398b0a91-plugins-config-volume\") pod \"payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6\" (UID: \"f0781fe8-13dc-4f23-9574-9967398b0a91\") " pod="openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6" Nov 03 16:42:25.702808 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.702774 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6560e01c-5031-49da-aea1-19d2df514d44-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d\" (UID: \"6560e01c-5031-49da-aea1-19d2df514d44\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d" Nov 03 16:42:25.736062 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.736020 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsw4p\" (UniqueName: \"kubernetes.io/projected/f0781fe8-13dc-4f23-9574-9967398b0a91-kube-api-access-bsw4p\") pod \"payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6\" (UID: \"f0781fe8-13dc-4f23-9574-9967398b0a91\") " pod="openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6" Nov 03 16:42:25.744108 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.744027 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6cmh\" (UniqueName: \"kubernetes.io/projected/6560e01c-5031-49da-aea1-19d2df514d44-kube-api-access-d6cmh\") pod \"payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d\" (UID: \"6560e01c-5031-49da-aea1-19d2df514d44\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d" Nov 03 16:42:25.752412 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.752135 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6" Nov 03 16:42:25.825333 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:25.823976 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d" Nov 03 16:42:26.174683 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:26.174630 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6"] Nov 03 16:42:26.177434 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:42:26.177402 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0781fe8_13dc_4f23_9574_9967398b0a91.slice/crio-da952f5775eb00db822feb6a27460d447d9611d17a6810d8c18099555e1ed5e5 WatchSource:0}: Error finding container da952f5775eb00db822feb6a27460d447d9611d17a6810d8c18099555e1ed5e5: Status 404 returned error can't find the container with id da952f5775eb00db822feb6a27460d447d9611d17a6810d8c18099555e1ed5e5 Nov 03 16:42:26.190967 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:42:26.190921 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6560e01c_5031_49da_aea1_19d2df514d44.slice/crio-fe60a3b23a3d5093dc4e63e62760496fa9ec77eb66ac3dcc116d03710febccf5 WatchSource:0}: Error finding container fe60a3b23a3d5093dc4e63e62760496fa9ec77eb66ac3dcc116d03710febccf5: Status 404 returned error can't find the container with id fe60a3b23a3d5093dc4e63e62760496fa9ec77eb66ac3dcc116d03710febccf5 Nov 03 16:42:26.192508 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:26.192486 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d"] Nov 03 16:42:27.160052 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:27.160023 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d" event={"ID":"6560e01c-5031-49da-aea1-19d2df514d44","Type":"ContainerStarted","Data":"ce0a3d14ba35b8b1bc831fa8763234d9095cbada0f982749f25b8cb804ab2c91"} Nov 03 16:42:27.160547 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:27.160076 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d" event={"ID":"6560e01c-5031-49da-aea1-19d2df514d44","Type":"ContainerStarted","Data":"fe60a3b23a3d5093dc4e63e62760496fa9ec77eb66ac3dcc116d03710febccf5"} Nov 03 16:42:27.160547 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:27.160100 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d" Nov 03 16:42:27.161844 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:27.161821 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6" event={"ID":"f0781fe8-13dc-4f23-9574-9967398b0a91","Type":"ContainerStarted","Data":"b0a74142c5f66592110f7d0cd75536871d9fb878e0ec39b7ee1a1b965e32692e"} Nov 03 16:42:27.161959 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:27.161867 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6" event={"ID":"f0781fe8-13dc-4f23-9574-9967398b0a91","Type":"ContainerStarted","Data":"da952f5775eb00db822feb6a27460d447d9611d17a6810d8c18099555e1ed5e5"} Nov 03 16:42:27.161959 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:27.161895 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6" Nov 03 16:42:27.206193 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:27.206137 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d" podStartSLOduration=2.206118971 podStartE2EDuration="2.206118971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:42:27.204628645 +0000 UTC m=+1690.793628613" watchObservedRunningTime="2026-11-03 16:42:27.206118971 +0000 UTC m=+1690.795118940" Nov 03 16:42:27.249822 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:27.249781 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6" podStartSLOduration=2.249770009 podStartE2EDuration="2.249770009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:42:27.249614029 +0000 UTC m=+1690.838613996" watchObservedRunningTime="2026-11-03 16:42:27.249770009 +0000 UTC m=+1690.838769977" Nov 03 16:42:30.131509 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:42:30.131473 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:42:31.488126 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:31.488095 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4"] Nov 03 16:42:31.488928 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:31.488893 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" podUID="d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" containerName="istio-proxy" containerID="cri-o://4254b24b4d387b8b283574032703df31f1cf632a9b7c78da9718024bc2f4dc08" gracePeriod=30 Nov 03 16:42:35.376971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:35.376928 4947 patch_prober.go:29] interesting pod/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:42:35.377438 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:35.376988 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" podUID="d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:42:35.586398 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:35.586363 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:42:36.808822 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.808780 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:42:36.911518 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.911483 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-kube-api-access-s7klr\" (UniqueName: \"kubernetes.io/projected/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-kube-api-access-s7klr\") pod \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " Nov 03 16:42:36.911738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.911587 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-podinfo\") pod \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " Nov 03 16:42:36.911738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.911664 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-workload-certs\") pod \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " Nov 03 16:42:36.911860 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.911795 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-workload-socket\") pod \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " Nov 03 16:42:36.911913 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.911873 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-envoy\") pod \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " Nov 03 16:42:36.911968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.911930 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-data\") pod \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " Nov 03 16:42:36.912019 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.911970 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-token\" (UniqueName: \"kubernetes.io/projected/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-token\") pod \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " Nov 03 16:42:36.912069 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.912052 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istiod-ca-cert\") pod \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " Nov 03 16:42:36.912440 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.912099 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-workload-socket" pod "d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" (UID: "d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:42:36.912440 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.912153 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-credential-socket\") pod \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\" (UID: \"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c\") " Nov 03 16:42:36.912440 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.912229 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-workload-certs" pod "d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" (UID: "d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:42:36.912440 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.912357 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-data" pod "d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" (UID: "d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:42:36.912829 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.912628 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-credential-socket" pod "d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" (UID: "d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:42:36.912829 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.912741 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istiod-ca-cert" pod "d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" (UID: "d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:42:36.913230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.913208 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:42:36.913388 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.913365 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:42:36.913388 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.913389 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:42:36.913585 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.913404 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:42:36.913585 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.913417 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:42:36.914340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.914305 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-kube-api-access-s7klr" pod "d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" (UID: "d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c"). InnerVolumeSpecName "kube-api-access-s7klr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:42:36.914910 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.914882 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-envoy" pod "d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" (UID: "d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:42:36.915063 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.915041 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-podinfo" pod "d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" (UID: "d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:42:36.915323 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:36.915307 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-token" pod "d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" (UID: "d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:42:37.015030 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:37.014999 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:42:37.015030 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:37.015028 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:42:37.015030 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:37.015041 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7klr\" (UniqueName: \"kubernetes.io/projected/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-kube-api-access-s7klr\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:42:37.015323 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:37.015055 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:42:37.262136 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:37.262103 4947 generic.go:356] "Generic (PLEG): container finished" podID="d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" containerID="4254b24b4d387b8b283574032703df31f1cf632a9b7c78da9718024bc2f4dc08" exitCode=0 Nov 03 16:42:37.262320 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:37.262159 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" Nov 03 16:42:37.262320 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:37.262188 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" event={"ID":"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c","Type":"ContainerDied","Data":"4254b24b4d387b8b283574032703df31f1cf632a9b7c78da9718024bc2f4dc08"} Nov 03 16:42:37.262320 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:37.262249 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4" event={"ID":"d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c","Type":"ContainerDied","Data":"9165c20a4115f4400a60dc6e1c4cec1bfd0ff40c89ba92c78c2b686016841cfc"} Nov 03 16:42:37.262320 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:37.262272 4947 scope.go:122] "RemoveContainer" containerID="4254b24b4d387b8b283574032703df31f1cf632a9b7c78da9718024bc2f4dc08" Nov 03 16:42:37.274020 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:37.273999 4947 scope.go:122] "RemoveContainer" containerID="bc2c45d2210c6b7acad5cfde81d9657646c80437105523dad5fe21285aba3d44" Nov 03 16:42:37.284495 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:37.284474 4947 scope.go:122] "RemoveContainer" containerID="4254b24b4d387b8b283574032703df31f1cf632a9b7c78da9718024bc2f4dc08" Nov 03 16:42:37.284800 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:42:37.284779 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4254b24b4d387b8b283574032703df31f1cf632a9b7c78da9718024bc2f4dc08\": container with ID starting with 4254b24b4d387b8b283574032703df31f1cf632a9b7c78da9718024bc2f4dc08 not found: ID does not exist" containerID="4254b24b4d387b8b283574032703df31f1cf632a9b7c78da9718024bc2f4dc08" Nov 03 16:42:37.284874 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:37.284809 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4254b24b4d387b8b283574032703df31f1cf632a9b7c78da9718024bc2f4dc08"} err="failed to get container status \"4254b24b4d387b8b283574032703df31f1cf632a9b7c78da9718024bc2f4dc08\": rpc error: code = NotFound desc = could not find container \"4254b24b4d387b8b283574032703df31f1cf632a9b7c78da9718024bc2f4dc08\": container with ID starting with 4254b24b4d387b8b283574032703df31f1cf632a9b7c78da9718024bc2f4dc08 not found: ID does not exist" Nov 03 16:42:37.284874 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:37.284826 4947 scope.go:122] "RemoveContainer" containerID="bc2c45d2210c6b7acad5cfde81d9657646c80437105523dad5fe21285aba3d44" Nov 03 16:42:37.285079 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:42:37.285062 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2c45d2210c6b7acad5cfde81d9657646c80437105523dad5fe21285aba3d44\": container with ID starting with bc2c45d2210c6b7acad5cfde81d9657646c80437105523dad5fe21285aba3d44 not found: ID does not exist" containerID="bc2c45d2210c6b7acad5cfde81d9657646c80437105523dad5fe21285aba3d44" Nov 03 16:42:37.285127 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:37.285086 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2c45d2210c6b7acad5cfde81d9657646c80437105523dad5fe21285aba3d44"} err="failed to get container status \"bc2c45d2210c6b7acad5cfde81d9657646c80437105523dad5fe21285aba3d44\": rpc error: code = NotFound desc = could not find container \"bc2c45d2210c6b7acad5cfde81d9657646c80437105523dad5fe21285aba3d44\": container with ID starting with bc2c45d2210c6b7acad5cfde81d9657646c80437105523dad5fe21285aba3d44 not found: ID does not exist" Nov 03 16:42:37.325020 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:37.324994 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4"] Nov 03 16:42:37.338649 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:37.338615 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9501ab-openshift-default-57c8959f65-v4qn4"] Nov 03 16:42:38.179052 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:38.179024 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6" Nov 03 16:42:38.179500 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:38.179104 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d" Nov 03 16:42:38.326828 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:38.326796 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps"] Nov 03 16:42:38.353740 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:38.353681 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n"] Nov 03 16:42:38.567657 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:38.567626 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" path="/var/lib/kubelet/pods/d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c/volumes" Nov 03 16:42:43.328915 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:43.328465 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps" podUID="5d560fb8-8a25-43a4-9327-0e59687c3bd9" containerName="payload-processing" containerID="cri-o://1a8925df267cd9a298fe5870275dece9de1546e03ff8265b981068bb2f917df2" gracePeriod=25 Nov 03 16:42:43.354766 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:43.354738 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n" podUID="f58e71f9-4452-4e6a-a70d-642ab5815865" containerName="payload-pre-processing" containerID="cri-o://df5bac6bc2c019ee6917e12f8acef5b2c454ec1f27367f8e884c3da9375c41aa" gracePeriod=25 Nov 03 16:42:43.830200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:43.830158 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps" Nov 03 16:42:43.884548 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:43.884520 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n" Nov 03 16:42:43.928337 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:43.928302 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d560fb8-8a25-43a4-9327-0e59687c3bd9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d560fb8-8a25-43a4-9327-0e59687c3bd9-plugins-config-volume\") pod \"5d560fb8-8a25-43a4-9327-0e59687c3bd9\" (UID: \"5d560fb8-8a25-43a4-9327-0e59687c3bd9\") " Nov 03 16:42:43.928551 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:43.928346 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58e71f9-4452-4e6a-a70d-642ab5815865-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58e71f9-4452-4e6a-a70d-642ab5815865-plugins-config-volume\") pod \"f58e71f9-4452-4e6a-a70d-642ab5815865\" (UID: \"f58e71f9-4452-4e6a-a70d-642ab5815865\") " Nov 03 16:42:43.928551 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:43.928490 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d560fb8-8a25-43a4-9327-0e59687c3bd9-kube-api-access-kkv28\" (UniqueName: \"kubernetes.io/projected/5d560fb8-8a25-43a4-9327-0e59687c3bd9-kube-api-access-kkv28\") pod \"5d560fb8-8a25-43a4-9327-0e59687c3bd9\" (UID: \"5d560fb8-8a25-43a4-9327-0e59687c3bd9\") " Nov 03 16:42:43.928740 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:43.928590 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58e71f9-4452-4e6a-a70d-642ab5815865-kube-api-access-5z68x\" (UniqueName: \"kubernetes.io/projected/f58e71f9-4452-4e6a-a70d-642ab5815865-kube-api-access-5z68x\") pod \"f58e71f9-4452-4e6a-a70d-642ab5815865\" (UID: \"f58e71f9-4452-4e6a-a70d-642ab5815865\") " Nov 03 16:42:43.928844 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:43.928816 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58e71f9-4452-4e6a-a70d-642ab5815865-plugins-config-volume" pod "f58e71f9-4452-4e6a-a70d-642ab5815865" (UID: "f58e71f9-4452-4e6a-a70d-642ab5815865"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:42:43.929304 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:43.929276 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58e71f9-4452-4e6a-a70d-642ab5815865-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:42:43.929459 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:43.929431 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d560fb8-8a25-43a4-9327-0e59687c3bd9-plugins-config-volume" pod "5d560fb8-8a25-43a4-9327-0e59687c3bd9" (UID: "5d560fb8-8a25-43a4-9327-0e59687c3bd9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:42:43.931248 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:43.931221 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d560fb8-8a25-43a4-9327-0e59687c3bd9-kube-api-access-kkv28" pod "5d560fb8-8a25-43a4-9327-0e59687c3bd9" (UID: "5d560fb8-8a25-43a4-9327-0e59687c3bd9"). InnerVolumeSpecName "kube-api-access-kkv28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:42:43.931618 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:43.931598 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58e71f9-4452-4e6a-a70d-642ab5815865-kube-api-access-5z68x" pod "f58e71f9-4452-4e6a-a70d-642ab5815865" (UID: "f58e71f9-4452-4e6a-a70d-642ab5815865"). InnerVolumeSpecName "kube-api-access-5z68x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:42:44.031250 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.031155 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d560fb8-8a25-43a4-9327-0e59687c3bd9-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:42:44.031250 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.031184 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkv28\" (UniqueName: \"kubernetes.io/projected/5d560fb8-8a25-43a4-9327-0e59687c3bd9-kube-api-access-kkv28\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:42:44.031250 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.031202 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z68x\" (UniqueName: \"kubernetes.io/projected/f58e71f9-4452-4e6a-a70d-642ab5815865-kube-api-access-5z68x\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:42:44.448565 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.448519 4947 generic.go:356] "Generic (PLEG): container finished" podID="5d560fb8-8a25-43a4-9327-0e59687c3bd9" containerID="1a8925df267cd9a298fe5870275dece9de1546e03ff8265b981068bb2f917df2" exitCode=0 Nov 03 16:42:44.449083 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.448580 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps" event={"ID":"5d560fb8-8a25-43a4-9327-0e59687c3bd9","Type":"ContainerDied","Data":"1a8925df267cd9a298fe5870275dece9de1546e03ff8265b981068bb2f917df2"} Nov 03 16:42:44.449083 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.448604 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps" Nov 03 16:42:44.449083 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.448680 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps" event={"ID":"5d560fb8-8a25-43a4-9327-0e59687c3bd9","Type":"ContainerDied","Data":"0d1f10b2e444d9a662323873cff6de2d5433bea8e62a4fedecaf8d59ce74f164"} Nov 03 16:42:44.449083 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.448745 4947 scope.go:122] "RemoveContainer" containerID="1a8925df267cd9a298fe5870275dece9de1546e03ff8265b981068bb2f917df2" Nov 03 16:42:44.450279 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.450256 4947 generic.go:356] "Generic (PLEG): container finished" podID="f58e71f9-4452-4e6a-a70d-642ab5815865" containerID="df5bac6bc2c019ee6917e12f8acef5b2c454ec1f27367f8e884c3da9375c41aa" exitCode=0 Nov 03 16:42:44.450417 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.450400 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n" event={"ID":"f58e71f9-4452-4e6a-a70d-642ab5815865","Type":"ContainerDied","Data":"df5bac6bc2c019ee6917e12f8acef5b2c454ec1f27367f8e884c3da9375c41aa"} Nov 03 16:42:44.450488 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.450447 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n" event={"ID":"f58e71f9-4452-4e6a-a70d-642ab5815865","Type":"ContainerDied","Data":"821d48b1bc812d027ae612aadebd43aaee324a3e44c8970d8b7c0b9d6a42c3b0"} Nov 03 16:42:44.450608 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.450581 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n" Nov 03 16:42:44.460922 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.460904 4947 scope.go:122] "RemoveContainer" containerID="1a8925df267cd9a298fe5870275dece9de1546e03ff8265b981068bb2f917df2" Nov 03 16:42:44.461164 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:42:44.461142 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a8925df267cd9a298fe5870275dece9de1546e03ff8265b981068bb2f917df2\": container with ID starting with 1a8925df267cd9a298fe5870275dece9de1546e03ff8265b981068bb2f917df2 not found: ID does not exist" containerID="1a8925df267cd9a298fe5870275dece9de1546e03ff8265b981068bb2f917df2" Nov 03 16:42:44.461217 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.461171 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a8925df267cd9a298fe5870275dece9de1546e03ff8265b981068bb2f917df2"} err="failed to get container status \"1a8925df267cd9a298fe5870275dece9de1546e03ff8265b981068bb2f917df2\": rpc error: code = NotFound desc = could not find container \"1a8925df267cd9a298fe5870275dece9de1546e03ff8265b981068bb2f917df2\": container with ID starting with 1a8925df267cd9a298fe5870275dece9de1546e03ff8265b981068bb2f917df2 not found: ID does not exist" Nov 03 16:42:44.461217 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.461193 4947 scope.go:122] "RemoveContainer" containerID="df5bac6bc2c019ee6917e12f8acef5b2c454ec1f27367f8e884c3da9375c41aa" Nov 03 16:42:44.470888 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.470871 4947 scope.go:122] "RemoveContainer" containerID="df5bac6bc2c019ee6917e12f8acef5b2c454ec1f27367f8e884c3da9375c41aa" Nov 03 16:42:44.471142 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:42:44.471121 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df5bac6bc2c019ee6917e12f8acef5b2c454ec1f27367f8e884c3da9375c41aa\": container with ID starting with df5bac6bc2c019ee6917e12f8acef5b2c454ec1f27367f8e884c3da9375c41aa not found: ID does not exist" containerID="df5bac6bc2c019ee6917e12f8acef5b2c454ec1f27367f8e884c3da9375c41aa" Nov 03 16:42:44.471219 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.471152 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5bac6bc2c019ee6917e12f8acef5b2c454ec1f27367f8e884c3da9375c41aa"} err="failed to get container status \"df5bac6bc2c019ee6917e12f8acef5b2c454ec1f27367f8e884c3da9375c41aa\": rpc error: code = NotFound desc = could not find container \"df5bac6bc2c019ee6917e12f8acef5b2c454ec1f27367f8e884c3da9375c41aa\": container with ID starting with df5bac6bc2c019ee6917e12f8acef5b2c454ec1f27367f8e884c3da9375c41aa not found: ID does not exist" Nov 03 16:42:44.526074 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.526029 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n"] Nov 03 16:42:44.535904 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.535873 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d89163-7f5df67b5f-vgk2n"] Nov 03 16:42:44.566830 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.566801 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58e71f9-4452-4e6a-a70d-642ab5815865" path="/var/lib/kubelet/pods/f58e71f9-4452-4e6a-a70d-642ab5815865/volumes" Nov 03 16:42:44.567209 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.567191 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps"] Nov 03 16:42:44.567248 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:44.567232 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d89163-54b99f7d67-r8hps"] Nov 03 16:42:46.567052 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:46.567019 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d560fb8-8a25-43a4-9327-0e59687c3bd9" path="/var/lib/kubelet/pods/5d560fb8-8a25-43a4-9327-0e59687c3bd9/volumes" Nov 03 16:42:48.615580 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:48.615382 4947 patch_prober.go:29] interesting pod/e2e-api-b-d89163-openshift-default-69879764b9-h8szp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": read tcp 10.132.0.2:46510->10.132.0.89:15021: read: connection reset by peer" start-of-body= Nov 03 16:42:48.615580 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:48.615447 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" podUID="0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": read tcp 10.132.0.2:46510->10.132.0.89:15021: read: connection reset by peer" Nov 03 16:42:49.540747 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:49.540712 4947 generic.go:356] "Generic (PLEG): container finished" podID="0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" containerID="cf985113eb07ee321b5f96576d58fe60cd3b6d258ff883be46a11242203c92ef" exitCode=0 Nov 03 16:42:49.540747 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:49.540724 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" event={"ID":"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef","Type":"ContainerDied","Data":"cf985113eb07ee321b5f96576d58fe60cd3b6d258ff883be46a11242203c92ef"} Nov 03 16:42:49.541123 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:49.541109 4947 scope.go:122] "RemoveContainer" containerID="cf985113eb07ee321b5f96576d58fe60cd3b6d258ff883be46a11242203c92ef" Nov 03 16:42:49.542511 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:49.542485 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:42:49.542600 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:49.542563 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:42:49.542638 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:49.542614 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:42:50.681720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:50.673471 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" event={"ID":"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef","Type":"ContainerStarted","Data":"e57574ecb44e8301d8ebc7a32f4ec6b72183c13eaba3ee3fedccb5c073d11fe1"} Nov 03 16:42:51.198722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:51.181772 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:42:51.198722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:51.181900 4947 patch_prober.go:29] interesting pod/e2e-api-b-d89163-openshift-default-69879764b9-h8szp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 16:42:51.198722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:51.181956 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" podUID="0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 16:42:52.181988 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:52.181953 4947 patch_prober.go:29] interesting pod/e2e-api-b-d89163-openshift-default-69879764b9-h8szp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 16:42:52.182367 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:52.182005 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" podUID="0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 16:42:53.182438 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:53.182400 4947 patch_prober.go:29] interesting pod/e2e-api-b-d89163-openshift-default-69879764b9-h8szp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 16:42:53.182890 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:53.182468 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" podUID="0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 16:42:54.182235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:54.182195 4947 patch_prober.go:29] interesting pod/e2e-api-b-d89163-openshift-default-69879764b9-h8szp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 16:42:54.182415 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:54.182264 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" podUID="0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 16:42:55.181775 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:55.181743 4947 patch_prober.go:29] interesting pod/e2e-api-b-d89163-openshift-default-69879764b9-h8szp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 16:42:55.182198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:55.181804 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" podUID="0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 16:42:55.820984 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:55.820949 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp"] Nov 03 16:42:55.821207 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:42:55.821176 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" podUID="0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" containerName="istio-proxy" containerID="cri-o://e57574ecb44e8301d8ebc7a32f4ec6b72183c13eaba3ee3fedccb5c073d11fe1" gracePeriod=30 Nov 03 16:43:00.073490 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.073459 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv"] Nov 03 16:43:00.074122 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.074106 4947 cpu_manager.go:414] "containerMap: removing container" podUID="d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" containerName="istio-proxy" Nov 03 16:43:00.074174 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.074123 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" containerName="istio-proxy" Nov 03 16:43:00.074174 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.074135 4947 cpu_manager.go:414] "containerMap: removing container" podUID="d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" containerName="istio-proxy" Nov 03 16:43:00.074174 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.074141 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" containerName="istio-proxy" Nov 03 16:43:00.074271 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.074177 4947 cpu_manager.go:414] "containerMap: removing container" podUID="f58e71f9-4452-4e6a-a70d-642ab5815865" containerName="payload-pre-processing" Nov 03 16:43:00.074271 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.074182 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58e71f9-4452-4e6a-a70d-642ab5815865" containerName="payload-pre-processing" Nov 03 16:43:00.074271 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.074198 4947 cpu_manager.go:414] "containerMap: removing container" podUID="5d560fb8-8a25-43a4-9327-0e59687c3bd9" containerName="payload-processing" Nov 03 16:43:00.074271 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.074203 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d560fb8-8a25-43a4-9327-0e59687c3bd9" containerName="payload-processing" Nov 03 16:43:00.074387 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.074288 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58e71f9-4452-4e6a-a70d-642ab5815865" containerName="payload-pre-processing" Nov 03 16:43:00.074387 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.074296 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d560fb8-8a25-43a4-9327-0e59687c3bd9" containerName="payload-processing" Nov 03 16:43:00.074387 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.074304 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" containerName="istio-proxy" Nov 03 16:43:00.074387 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.074310 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="d552ecdb-c3a5-4ee8-ad18-fe6809cbf03c" containerName="istio-proxy" Nov 03 16:43:00.077369 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.077354 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.094557 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.094529 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4e3be7-openshift-default-dockercfg-rrmfn\"" Nov 03 16:43:00.114690 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.114658 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv"] Nov 03 16:43:00.142548 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:43:00.142506 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:43:00.227720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.227277 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zhd9\" (UniqueName: \"kubernetes.io/projected/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-kube-api-access-7zhd9\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.227720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.227385 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-workload-socket\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.227720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.227422 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-podinfo\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.227720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.227501 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-envoy\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.227720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.227526 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istiod-ca-cert\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.227720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.227662 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-token\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.268721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.262915 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-workload-certs\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.268721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.262968 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-credential-socket\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.268721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.263045 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-data\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.374770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.368715 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-workload-certs\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.374770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.368785 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-credential-socket\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.374770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.368856 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-data\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.374770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.369051 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zhd9\" (UniqueName: \"kubernetes.io/projected/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-kube-api-access-7zhd9\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.374770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.369085 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-workload-socket\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.374770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.369114 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-podinfo\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.374770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.369190 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-envoy\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.374770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.369216 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istiod-ca-cert\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.374770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.369352 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-token\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.374770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.370221 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-workload-certs\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.374770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.370589 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-credential-socket\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.374770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.371033 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-data\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.385272 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.385238 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istiod-ca-cert\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.385632 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.385553 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-workload-socket\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.387977 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.387933 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-envoy\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.397570 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.397336 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-token\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.400722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.400624 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zhd9\" (UniqueName: \"kubernetes.io/projected/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-kube-api-access-7zhd9\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.418673 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.416070 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-podinfo\") pod \"e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:00.690737 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:00.690540 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:01.018451 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.018418 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv"] Nov 03 16:43:01.018874 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:43:01.018848 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0cbf00_cde2_4e77_a8f1_5bf936b6a0f3.slice/crio-b3561a375be564ed8a7dadb0b88c41f3ed195c94f0184ec37d76a8d0afcc96cb WatchSource:0}: Error finding container b3561a375be564ed8a7dadb0b88c41f3ed195c94f0184ec37d76a8d0afcc96cb: Status 404 returned error can't find the container with id b3561a375be564ed8a7dadb0b88c41f3ed195c94f0184ec37d76a8d0afcc96cb Nov 03 16:43:01.021438 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.021398 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:43:01.021543 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.021470 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:43:01.021543 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.021498 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:43:01.492720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.491990 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:43:01.798537 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.798500 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" event={"ID":"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3","Type":"ContainerStarted","Data":"5dba12412ca78a005f1d5e2cb6829c1d43d13f2f0af966da3d7860b6b2e17266"} Nov 03 16:43:01.798753 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.798560 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" event={"ID":"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3","Type":"ContainerStarted","Data":"b3561a375be564ed8a7dadb0b88c41f3ed195c94f0184ec37d76a8d0afcc96cb"} Nov 03 16:43:01.800803 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.800777 4947 generic.go:356] "Generic (PLEG): container finished" podID="0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" containerID="e57574ecb44e8301d8ebc7a32f4ec6b72183c13eaba3ee3fedccb5c073d11fe1" exitCode=0 Nov 03 16:43:01.800959 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.800853 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" event={"ID":"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef","Type":"ContainerDied","Data":"e57574ecb44e8301d8ebc7a32f4ec6b72183c13eaba3ee3fedccb5c073d11fe1"} Nov 03 16:43:01.800959 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.800917 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" event={"ID":"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef","Type":"ContainerDied","Data":"8a6fe6ed4f3d6ff1aa2a1ca7103ee364400aa36593c1bbd333859b0ffe79ebb1"} Nov 03 16:43:01.800959 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.800941 4947 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a6fe6ed4f3d6ff1aa2a1ca7103ee364400aa36593c1bbd333859b0ffe79ebb1" Nov 03 16:43:01.800959 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.800959 4947 scope.go:122] "RemoveContainer" containerID="cf985113eb07ee321b5f96576d58fe60cd3b6d258ff883be46a11242203c92ef" Nov 03 16:43:01.816544 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.816522 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:43:01.862116 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.862062 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" podStartSLOduration=1.862048296 podStartE2EDuration="1.862048296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:01.860241218 +0000 UTC m=+1725.449241186" watchObservedRunningTime="2026-11-03 16:43:01.862048296 +0000 UTC m=+1725.451048263" Nov 03 16:43:01.927303 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.927265 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-envoy\") pod \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " Nov 03 16:43:01.927489 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.927353 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-data\") pod \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " Nov 03 16:43:01.927489 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.927377 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-workload-socket\") pod \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " Nov 03 16:43:01.927608 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.927504 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-credential-socket\") pod \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " Nov 03 16:43:01.927608 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.927551 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-workload-certs\") pod \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " Nov 03 16:43:01.927737 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.927604 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-token\" (UniqueName: \"kubernetes.io/projected/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-token\") pod \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " Nov 03 16:43:01.927737 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.927654 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-data" pod "0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" (UID: "0b2aa7e1-3ff0-469b-9cb1-418031bf90ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:43:01.927737 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.927665 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-podinfo\") pod \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " Nov 03 16:43:01.927897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.927762 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istiod-ca-cert\") pod \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " Nov 03 16:43:01.927897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.927794 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-workload-socket" pod "0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" (UID: "0b2aa7e1-3ff0-469b-9cb1-418031bf90ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:43:01.927897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.927863 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-workload-certs" pod "0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" (UID: "0b2aa7e1-3ff0-469b-9cb1-418031bf90ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:43:01.927897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.927869 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-kube-api-access-596ft\" (UniqueName: \"kubernetes.io/projected/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-kube-api-access-596ft\") pod \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\" (UID: \"0b2aa7e1-3ff0-469b-9cb1-418031bf90ef\") " Nov 03 16:43:01.927897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.927886 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-credential-socket" pod "0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" (UID: "0b2aa7e1-3ff0-469b-9cb1-418031bf90ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:43:01.928448 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.928416 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istiod-ca-cert" pod "0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" (UID: "0b2aa7e1-3ff0-469b-9cb1-418031bf90ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:43:01.928581 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.928567 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:43:01.928628 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.928587 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:43:01.928628 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.928600 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:43:01.928628 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.928613 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:43:01.928749 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.928628 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:43:01.930456 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.930425 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-envoy" pod "0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" (UID: "0b2aa7e1-3ff0-469b-9cb1-418031bf90ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:43:01.930908 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.930869 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-token" pod "0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" (UID: "0b2aa7e1-3ff0-469b-9cb1-418031bf90ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:43:01.931005 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.930933 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-podinfo" pod "0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" (UID: "0b2aa7e1-3ff0-469b-9cb1-418031bf90ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:43:01.931066 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:01.931053 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-kube-api-access-596ft" pod "0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" (UID: "0b2aa7e1-3ff0-469b-9cb1-418031bf90ef"). InnerVolumeSpecName "kube-api-access-596ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:43:02.087483 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:02.086067 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-596ft\" (UniqueName: \"kubernetes.io/projected/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-kube-api-access-596ft\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:43:02.087483 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:02.086111 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:43:02.087483 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:02.086127 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:43:02.087483 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:02.086140 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:43:02.693194 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:02.691937 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:02.726027 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:02.725998 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:02.823322 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:02.823279 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp" Nov 03 16:43:02.823914 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:02.823892 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:02.827268 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:02.827240 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:02.970456 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:02.970363 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp"] Nov 03 16:43:02.978579 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:02.978546 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d89163-openshift-default-69879764b9-h8szp"] Nov 03 16:43:04.581236 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:04.581202 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" path="/var/lib/kubelet/pods/0b2aa7e1-3ff0-469b-9cb1-418031bf90ef/volumes" Nov 03 16:43:06.341505 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.341474 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt"] Nov 03 16:43:06.342099 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.342084 4947 cpu_manager.go:414] "containerMap: removing container" podUID="0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" containerName="istio-proxy" Nov 03 16:43:06.342099 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.342099 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" containerName="istio-proxy" Nov 03 16:43:06.342185 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.342109 4947 cpu_manager.go:414] "containerMap: removing container" podUID="0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" containerName="istio-proxy" Nov 03 16:43:06.342185 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.342114 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" containerName="istio-proxy" Nov 03 16:43:06.342250 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.342234 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" containerName="istio-proxy" Nov 03 16:43:06.342250 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.342245 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b2aa7e1-3ff0-469b-9cb1-418031bf90ef" containerName="istio-proxy" Nov 03 16:43:06.346611 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.346593 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt" Nov 03 16:43:06.366636 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.366605 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt"] Nov 03 16:43:06.376115 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.376087 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4e3be7-dockercfg-vmmr4\"" Nov 03 16:43:06.376290 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.376179 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4e3be7\"" Nov 03 16:43:06.413003 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.412970 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw"] Nov 03 16:43:06.418426 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.418401 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw" Nov 03 16:43:06.419609 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.419589 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bed0898-f1a8-47ed-b47f-f6df2b263d74-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt\" (UID: \"4bed0898-f1a8-47ed-b47f-f6df2b263d74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt" Nov 03 16:43:06.419790 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.419771 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgwr7\" (UniqueName: \"kubernetes.io/projected/4bed0898-f1a8-47ed-b47f-f6df2b263d74-kube-api-access-mgwr7\") pod \"payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt\" (UID: \"4bed0898-f1a8-47ed-b47f-f6df2b263d74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt" Nov 03 16:43:06.438275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.438246 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw"] Nov 03 16:43:06.593777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.584637 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bed0898-f1a8-47ed-b47f-f6df2b263d74-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt\" (UID: \"4bed0898-f1a8-47ed-b47f-f6df2b263d74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt" Nov 03 16:43:06.598843 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.595391 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bed0898-f1a8-47ed-b47f-f6df2b263d74-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt\" (UID: \"4bed0898-f1a8-47ed-b47f-f6df2b263d74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt" Nov 03 16:43:06.599145 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.596014 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgwr7\" (UniqueName: \"kubernetes.io/projected/4bed0898-f1a8-47ed-b47f-f6df2b263d74-kube-api-access-mgwr7\") pod \"payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt\" (UID: \"4bed0898-f1a8-47ed-b47f-f6df2b263d74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt" Nov 03 16:43:06.599341 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.599321 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw\" (UID: \"c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf\") " pod="openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw" Nov 03 16:43:06.599504 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.599489 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckbll\" (UniqueName: \"kubernetes.io/projected/c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf-kube-api-access-ckbll\") pod \"payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw\" (UID: \"c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf\") " pod="openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw" Nov 03 16:43:06.655948 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.638422 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgwr7\" (UniqueName: \"kubernetes.io/projected/4bed0898-f1a8-47ed-b47f-f6df2b263d74-kube-api-access-mgwr7\") pod \"payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt\" (UID: \"4bed0898-f1a8-47ed-b47f-f6df2b263d74\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt" Nov 03 16:43:06.675407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.675096 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt" Nov 03 16:43:06.793370 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.793331 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw\" (UID: \"c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf\") " pod="openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw" Nov 03 16:43:06.793663 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.793639 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckbll\" (UniqueName: \"kubernetes.io/projected/c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf-kube-api-access-ckbll\") pod \"payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw\" (UID: \"c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf\") " pod="openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw" Nov 03 16:43:06.838788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.794734 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw\" (UID: \"c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf\") " pod="openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw" Nov 03 16:43:06.864893 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.861890 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckbll\" (UniqueName: \"kubernetes.io/projected/c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf-kube-api-access-ckbll\") pod \"payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw\" (UID: \"c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf\") " pod="openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw" Nov 03 16:43:06.864893 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.862896 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b"] Nov 03 16:43:06.864893 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:06.862937 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx"] Nov 03 16:43:07.073720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:07.069722 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw" Nov 03 16:43:07.178109 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:07.178081 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt"] Nov 03 16:43:07.513754 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:07.512854 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw"] Nov 03 16:43:07.901361 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:07.901326 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt" event={"ID":"4bed0898-f1a8-47ed-b47f-f6df2b263d74","Type":"ContainerStarted","Data":"5406fa4dc12ea35b734b4f950169f95b09c59eb7b1f674afb301e962dcf61aa1"} Nov 03 16:43:07.901534 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:07.901381 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt" event={"ID":"4bed0898-f1a8-47ed-b47f-f6df2b263d74","Type":"ContainerStarted","Data":"c8c09832e27fb9aae0f3abf085efcc0486be509320b09853d0fdef20641da4fb"} Nov 03 16:43:07.901534 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:07.901408 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt" Nov 03 16:43:07.903029 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:07.903007 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw" event={"ID":"c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf","Type":"ContainerStarted","Data":"1c6609f82934dcd9d23145201785f2a22e5a9ec7dc2159eb59468028d55debfc"} Nov 03 16:43:07.903129 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:07.903050 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw" event={"ID":"c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf","Type":"ContainerStarted","Data":"e60b4c19ecf34122e8d9ab06c8118202710dec948ad79d4267e9a06f62964840"} Nov 03 16:43:07.903129 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:07.903086 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw" Nov 03 16:43:07.946789 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:07.946689 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt" podStartSLOduration=1.946673047 podStartE2EDuration="1.946673047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:07.945008935 +0000 UTC m=+1731.534008903" watchObservedRunningTime="2026-11-03 16:43:07.946673047 +0000 UTC m=+1731.535673045" Nov 03 16:43:07.989589 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:07.989524 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw" podStartSLOduration=1.989509067 podStartE2EDuration="1.989509067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:07.988264721 +0000 UTC m=+1731.577264690" watchObservedRunningTime="2026-11-03 16:43:07.989509067 +0000 UTC m=+1731.578509033" Nov 03 16:43:11.864107 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:11.864069 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" podUID="9158c889-3243-4483-a481-3b41f97f0350" containerName="payload-processing" containerID="cri-o://471d258dfb88ae032f47b3672f55100023e37d58a3abefa08c02587a0cb72f5d" gracePeriod=25 Nov 03 16:43:11.864538 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:11.864077 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx" podUID="5af158d1-c8f3-42ff-89f6-cac3af71192b" containerName="payload-pre-processing" containerID="cri-o://2bd0f09e102e27fa99c1d9a05cc286e47645264123e55e17c5469b7759c3bbb6" gracePeriod=25 Nov 03 16:43:12.139589 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:12.139567 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx" Nov 03 16:43:12.212576 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:12.212546 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af158d1-c8f3-42ff-89f6-cac3af71192b-kube-api-access-rvctf\" (UniqueName: \"kubernetes.io/projected/5af158d1-c8f3-42ff-89f6-cac3af71192b-kube-api-access-rvctf\") pod \"5af158d1-c8f3-42ff-89f6-cac3af71192b\" (UID: \"5af158d1-c8f3-42ff-89f6-cac3af71192b\") " Nov 03 16:43:12.212790 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:12.212624 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5af158d1-c8f3-42ff-89f6-cac3af71192b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af158d1-c8f3-42ff-89f6-cac3af71192b-plugins-config-volume\") pod \"5af158d1-c8f3-42ff-89f6-cac3af71192b\" (UID: \"5af158d1-c8f3-42ff-89f6-cac3af71192b\") " Nov 03 16:43:12.213036 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:12.213011 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af158d1-c8f3-42ff-89f6-cac3af71192b-plugins-config-volume" pod "5af158d1-c8f3-42ff-89f6-cac3af71192b" (UID: "5af158d1-c8f3-42ff-89f6-cac3af71192b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:43:12.213411 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:12.213390 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5af158d1-c8f3-42ff-89f6-cac3af71192b-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:43:12.215075 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:12.215050 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af158d1-c8f3-42ff-89f6-cac3af71192b-kube-api-access-rvctf" pod "5af158d1-c8f3-42ff-89f6-cac3af71192b" (UID: "5af158d1-c8f3-42ff-89f6-cac3af71192b"). InnerVolumeSpecName "kube-api-access-rvctf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:43:12.315684 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:12.315653 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvctf\" (UniqueName: \"kubernetes.io/projected/5af158d1-c8f3-42ff-89f6-cac3af71192b-kube-api-access-rvctf\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:43:12.936519 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:12.936486 4947 generic.go:356] "Generic (PLEG): container finished" podID="5af158d1-c8f3-42ff-89f6-cac3af71192b" containerID="2bd0f09e102e27fa99c1d9a05cc286e47645264123e55e17c5469b7759c3bbb6" exitCode=0 Nov 03 16:43:12.937046 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:12.936530 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx" Nov 03 16:43:12.937046 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:12.936566 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx" event={"ID":"5af158d1-c8f3-42ff-89f6-cac3af71192b","Type":"ContainerDied","Data":"2bd0f09e102e27fa99c1d9a05cc286e47645264123e55e17c5469b7759c3bbb6"} Nov 03 16:43:12.937046 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:12.936621 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx" event={"ID":"5af158d1-c8f3-42ff-89f6-cac3af71192b","Type":"ContainerDied","Data":"47145e27f31c250138404fee0f776e52aba1afea4084d990b3b2ca3230233fdf"} Nov 03 16:43:12.937046 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:12.936642 4947 scope.go:122] "RemoveContainer" containerID="2bd0f09e102e27fa99c1d9a05cc286e47645264123e55e17c5469b7759c3bbb6" Nov 03 16:43:12.947483 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:12.947458 4947 scope.go:122] "RemoveContainer" containerID="2bd0f09e102e27fa99c1d9a05cc286e47645264123e55e17c5469b7759c3bbb6" Nov 03 16:43:12.947806 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:43:12.947780 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd0f09e102e27fa99c1d9a05cc286e47645264123e55e17c5469b7759c3bbb6\": container with ID starting with 2bd0f09e102e27fa99c1d9a05cc286e47645264123e55e17c5469b7759c3bbb6 not found: ID does not exist" containerID="2bd0f09e102e27fa99c1d9a05cc286e47645264123e55e17c5469b7759c3bbb6" Nov 03 16:43:12.947873 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:12.947811 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd0f09e102e27fa99c1d9a05cc286e47645264123e55e17c5469b7759c3bbb6"} err="failed to get container status \"2bd0f09e102e27fa99c1d9a05cc286e47645264123e55e17c5469b7759c3bbb6\": rpc error: code = NotFound desc = could not find container \"2bd0f09e102e27fa99c1d9a05cc286e47645264123e55e17c5469b7759c3bbb6\": container with ID starting with 2bd0f09e102e27fa99c1d9a05cc286e47645264123e55e17c5469b7759c3bbb6 not found: ID does not exist" Nov 03 16:43:12.996624 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:12.996598 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx"] Nov 03 16:43:13.010939 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:13.010917 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-c6dqx"] Nov 03 16:43:14.565613 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:14.565566 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af158d1-c8f3-42ff-89f6-cac3af71192b" path="/var/lib/kubelet/pods/5af158d1-c8f3-42ff-89f6-cac3af71192b/volumes" Nov 03 16:43:17.801673 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:17.801633 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" podUID="9158c889-3243-4483-a481-3b41f97f0350" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:43:17.887818 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:17.887492 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt"] Nov 03 16:43:17.890731 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:17.889172 4947 cpu_manager.go:414] "containerMap: removing container" podUID="5af158d1-c8f3-42ff-89f6-cac3af71192b" containerName="payload-pre-processing" Nov 03 16:43:17.890731 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:17.889198 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af158d1-c8f3-42ff-89f6-cac3af71192b" containerName="payload-pre-processing" Nov 03 16:43:17.890731 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:17.889594 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af158d1-c8f3-42ff-89f6-cac3af71192b" containerName="payload-pre-processing" Nov 03 16:43:17.900091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:17.900061 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:17.922296 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:17.922265 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bffe15-openshift-default-dockercfg-6g6hn\"" Nov 03 16:43:17.964432 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:17.964398 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt"] Nov 03 16:43:17.977151 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:17.977114 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-data\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:17.977330 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:17.977163 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istiod-ca-cert\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:17.977330 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:17.977267 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-token\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:17.977330 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:17.977318 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-workload-socket\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:17.977453 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:17.977429 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75bbk\" (UniqueName: \"kubernetes.io/projected/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-kube-api-access-75bbk\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:17.977532 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:17.977513 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-credential-socket\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:17.978197 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:17.977557 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-podinfo\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:17.978197 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:17.977764 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-workload-certs\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:17.978197 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:17.977798 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-envoy\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.080336 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.079827 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-workload-socket\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.080610 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.080586 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75bbk\" (UniqueName: \"kubernetes.io/projected/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-kube-api-access-75bbk\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.080800 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.080774 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-credential-socket\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.080957 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.080940 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-podinfo\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.081187 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.081171 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-workload-certs\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.081310 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.081294 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-envoy\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.081607 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.081590 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-data\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.081744 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.081728 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istiod-ca-cert\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.081884 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.081870 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-token\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.082013 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.081987 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-workload-certs\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.082166 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.080404 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-workload-socket\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.083650 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.083624 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istiod-ca-cert\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.083882 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.081237 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-credential-socket\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.084143 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.084121 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-data\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.114721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.092179 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-podinfo\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.123724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.120546 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-envoy\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.132420 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.132368 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-token\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.194441 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.194411 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75bbk\" (UniqueName: \"kubernetes.io/projected/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-kube-api-access-75bbk\") pod \"e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.221508 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.221040 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:18.733969 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:43:18.733936 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595b44f5_49e0_4cd7_bd8f_83a72f6f426c.slice/crio-97f0f65904aa8d12ad5876a1d83a739625e6e219e4056ef31f81a35dbb1da99e WatchSource:0}: Error finding container 97f0f65904aa8d12ad5876a1d83a739625e6e219e4056ef31f81a35dbb1da99e: Status 404 returned error can't find the container with id 97f0f65904aa8d12ad5876a1d83a739625e6e219e4056ef31f81a35dbb1da99e Nov 03 16:43:18.734952 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.734926 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt"] Nov 03 16:43:18.736764 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.736669 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:43:18.736868 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.736815 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:43:18.736868 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.736849 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:43:18.942724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.938143 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt" Nov 03 16:43:18.942724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.940874 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw" Nov 03 16:43:18.998484 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.998453 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" event={"ID":"595b44f5-49e0-4cd7-bd8f-83a72f6f426c","Type":"ContainerStarted","Data":"9942e727094dd7306b774fa7a1b12ae4965247ec53b3428af1234165a67c53b2"} Nov 03 16:43:18.998736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:18.998715 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" event={"ID":"595b44f5-49e0-4cd7-bd8f-83a72f6f426c","Type":"ContainerStarted","Data":"97f0f65904aa8d12ad5876a1d83a739625e6e219e4056ef31f81a35dbb1da99e"} Nov 03 16:43:19.094171 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:19.094122 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" podStartSLOduration=2.094107314 podStartE2EDuration="2.094107314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:19.093912772 +0000 UTC m=+1742.682912740" watchObservedRunningTime="2026-11-03 16:43:19.094107314 +0000 UTC m=+1742.683107282" Nov 03 16:43:19.239320 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:19.239238 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:19.248208 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:19.247836 4947 patch_prober.go:29] interesting pod/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 16:43:19.248208 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:19.247919 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" podUID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 16:43:20.225843 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:20.225813 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:21.012028 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:21.012000 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:21.012994 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:21.012971 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:43:23.404074 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.404040 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8"] Nov 03 16:43:23.408511 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.408496 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8" Nov 03 16:43:23.431673 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.431644 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8"] Nov 03 16:43:23.438212 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.438177 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-bffe15-dockercfg-8jz62\"" Nov 03 16:43:23.438354 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.438228 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-bffe15\"" Nov 03 16:43:23.674989 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.674898 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgjth\" (UniqueName: \"kubernetes.io/projected/1548c8a6-c110-4168-9568-5adcc07df255-kube-api-access-mgjth\") pod \"payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8\" (UID: \"1548c8a6-c110-4168-9568-5adcc07df255\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8" Nov 03 16:43:23.675752 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.675679 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1548c8a6-c110-4168-9568-5adcc07df255-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8\" (UID: \"1548c8a6-c110-4168-9568-5adcc07df255\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8" Nov 03 16:43:23.700835 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.700768 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m"] Nov 03 16:43:23.770959 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.770927 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m"] Nov 03 16:43:23.779875 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.772430 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m" Nov 03 16:43:23.784546 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.784515 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96672632-057a-4ca2-ae04-f5d429f470a7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bffe15-ff48449d5-t955m\" (UID: \"96672632-057a-4ca2-ae04-f5d429f470a7\") " pod="openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m" Nov 03 16:43:23.784883 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.784859 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgjth\" (UniqueName: \"kubernetes.io/projected/1548c8a6-c110-4168-9568-5adcc07df255-kube-api-access-mgjth\") pod \"payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8\" (UID: \"1548c8a6-c110-4168-9568-5adcc07df255\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8" Nov 03 16:43:23.785081 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.785054 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f29q\" (UniqueName: \"kubernetes.io/projected/96672632-057a-4ca2-ae04-f5d429f470a7-kube-api-access-9f29q\") pod \"payload-processing-e2e-api-a-bffe15-ff48449d5-t955m\" (UID: \"96672632-057a-4ca2-ae04-f5d429f470a7\") " pod="openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m" Nov 03 16:43:23.785295 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.785277 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1548c8a6-c110-4168-9568-5adcc07df255-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8\" (UID: \"1548c8a6-c110-4168-9568-5adcc07df255\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8" Nov 03 16:43:23.790634 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.790601 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1548c8a6-c110-4168-9568-5adcc07df255-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8\" (UID: \"1548c8a6-c110-4168-9568-5adcc07df255\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8" Nov 03 16:43:23.823321 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.823294 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgjth\" (UniqueName: \"kubernetes.io/projected/1548c8a6-c110-4168-9568-5adcc07df255-kube-api-access-mgjth\") pod \"payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8\" (UID: \"1548c8a6-c110-4168-9568-5adcc07df255\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8" Nov 03 16:43:23.886357 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.886321 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f29q\" (UniqueName: \"kubernetes.io/projected/96672632-057a-4ca2-ae04-f5d429f470a7-kube-api-access-9f29q\") pod \"payload-processing-e2e-api-a-bffe15-ff48449d5-t955m\" (UID: \"96672632-057a-4ca2-ae04-f5d429f470a7\") " pod="openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m" Nov 03 16:43:23.886622 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.886603 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96672632-057a-4ca2-ae04-f5d429f470a7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bffe15-ff48449d5-t955m\" (UID: \"96672632-057a-4ca2-ae04-f5d429f470a7\") " pod="openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m" Nov 03 16:43:23.887325 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.887299 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96672632-057a-4ca2-ae04-f5d429f470a7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bffe15-ff48449d5-t955m\" (UID: \"96672632-057a-4ca2-ae04-f5d429f470a7\") " pod="openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m" Nov 03 16:43:23.911347 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:23.911315 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f29q\" (UniqueName: \"kubernetes.io/projected/96672632-057a-4ca2-ae04-f5d429f470a7-kube-api-access-9f29q\") pod \"payload-processing-e2e-api-a-bffe15-ff48449d5-t955m\" (UID: \"96672632-057a-4ca2-ae04-f5d429f470a7\") " pod="openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m" Nov 03 16:43:24.022710 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:24.022674 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8" Nov 03 16:43:24.111833 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:24.111804 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m" Nov 03 16:43:24.355810 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:24.355779 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8"] Nov 03 16:43:24.355977 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:43:24.355939 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1548c8a6_c110_4168_9568_5adcc07df255.slice/crio-d387bc8571ce0ad7bba1985aa1fbcfe1879157ba7673322210ec3cc76e877992 WatchSource:0}: Error finding container d387bc8571ce0ad7bba1985aa1fbcfe1879157ba7673322210ec3cc76e877992: Status 404 returned error can't find the container with id d387bc8571ce0ad7bba1985aa1fbcfe1879157ba7673322210ec3cc76e877992 Nov 03 16:43:24.477727 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:43:24.477682 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96672632_057a_4ca2_ae04_f5d429f470a7.slice/crio-a8f4800f43a9f74be270e96224afc15691eec01fe0309e60e591eb570c9a6c32 WatchSource:0}: Error finding container a8f4800f43a9f74be270e96224afc15691eec01fe0309e60e591eb570c9a6c32: Status 404 returned error can't find the container with id a8f4800f43a9f74be270e96224afc15691eec01fe0309e60e591eb570c9a6c32 Nov 03 16:43:24.478123 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:24.478102 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m"] Nov 03 16:43:25.042588 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:25.042549 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m" event={"ID":"96672632-057a-4ca2-ae04-f5d429f470a7","Type":"ContainerStarted","Data":"4f9b185b06d1c5750e06f61788ed9836feac27a33048134b8f50b2cda956cb38"} Nov 03 16:43:25.042806 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:25.042625 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m" event={"ID":"96672632-057a-4ca2-ae04-f5d429f470a7","Type":"ContainerStarted","Data":"a8f4800f43a9f74be270e96224afc15691eec01fe0309e60e591eb570c9a6c32"} Nov 03 16:43:25.042806 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:25.042722 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m" Nov 03 16:43:25.044295 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:25.044259 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8" event={"ID":"1548c8a6-c110-4168-9568-5adcc07df255","Type":"ContainerStarted","Data":"fe3693c109056f38f582e883703a09beeb5548a0b753704b738c84f703ff19ac"} Nov 03 16:43:25.044415 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:25.044304 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8" event={"ID":"1548c8a6-c110-4168-9568-5adcc07df255","Type":"ContainerStarted","Data":"d387bc8571ce0ad7bba1985aa1fbcfe1879157ba7673322210ec3cc76e877992"} Nov 03 16:43:25.044415 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:25.044327 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8" Nov 03 16:43:25.102965 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:25.102914 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m" podStartSLOduration=2.102899088 podStartE2EDuration="2.102899088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:25.102092392 +0000 UTC m=+1748.691092351" watchObservedRunningTime="2026-11-03 16:43:25.102899088 +0000 UTC m=+1748.691899055" Nov 03 16:43:25.149924 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:25.149867 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8" podStartSLOduration=2.149849322 podStartE2EDuration="2.149849322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:25.148177884 +0000 UTC m=+1748.737177851" watchObservedRunningTime="2026-11-03 16:43:25.149849322 +0000 UTC m=+1748.738849290" Nov 03 16:43:27.781788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:27.775396 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:43:27.873814 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:27.873754 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" podUID="9158c889-3243-4483-a481-3b41f97f0350" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 16:43:30.274724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:30.273944 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 16:43:30.302724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:30.301966 4947 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:30.323722 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:43:30.315898 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:43:30.617779 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:30.617058 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:30.988783 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:30.980951 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:31.016468 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:31.007837 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:31.058727 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:31.057776 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:43:32.291804 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:32.291536 4947 generic.go:356] "Generic (PLEG): container finished" podID="9158c889-3243-4483-a481-3b41f97f0350" containerID="471d258dfb88ae032f47b3672f55100023e37d58a3abefa08c02587a0cb72f5d" exitCode=0 Nov 03 16:43:32.291804 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:32.291720 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" event={"ID":"9158c889-3243-4483-a481-3b41f97f0350","Type":"ContainerDied","Data":"471d258dfb88ae032f47b3672f55100023e37d58a3abefa08c02587a0cb72f5d"} Nov 03 16:43:32.377793 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:32.377770 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" Nov 03 16:43:32.480207 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:32.480167 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9158c889-3243-4483-a481-3b41f97f0350-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9158c889-3243-4483-a481-3b41f97f0350-plugins-config-volume\") pod \"9158c889-3243-4483-a481-3b41f97f0350\" (UID: \"9158c889-3243-4483-a481-3b41f97f0350\") " Nov 03 16:43:32.480540 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:32.480516 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9158c889-3243-4483-a481-3b41f97f0350-kube-api-access-ztj4v\" (UniqueName: \"kubernetes.io/projected/9158c889-3243-4483-a481-3b41f97f0350-kube-api-access-ztj4v\") pod \"9158c889-3243-4483-a481-3b41f97f0350\" (UID: \"9158c889-3243-4483-a481-3b41f97f0350\") " Nov 03 16:43:32.480671 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:32.480583 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9158c889-3243-4483-a481-3b41f97f0350-plugins-config-volume" pod "9158c889-3243-4483-a481-3b41f97f0350" (UID: "9158c889-3243-4483-a481-3b41f97f0350"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:43:32.481072 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:32.481049 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9158c889-3243-4483-a481-3b41f97f0350-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:43:32.482799 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:32.482771 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9158c889-3243-4483-a481-3b41f97f0350-kube-api-access-ztj4v" pod "9158c889-3243-4483-a481-3b41f97f0350" (UID: "9158c889-3243-4483-a481-3b41f97f0350"). InnerVolumeSpecName "kube-api-access-ztj4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:43:32.582353 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:32.582326 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztj4v\" (UniqueName: \"kubernetes.io/projected/9158c889-3243-4483-a481-3b41f97f0350-kube-api-access-ztj4v\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:43:33.323948 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:33.323902 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" Nov 03 16:43:33.324540 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:33.323899 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b" event={"ID":"9158c889-3243-4483-a481-3b41f97f0350","Type":"ContainerDied","Data":"4701198e237f67e296b16dd178f9ca1b3e5cbdeb33f14d4a366270ec65435e03"} Nov 03 16:43:33.324540 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:33.324056 4947 scope.go:122] "RemoveContainer" containerID="471d258dfb88ae032f47b3672f55100023e37d58a3abefa08c02587a0cb72f5d" Nov 03 16:43:33.381284 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:33.381253 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b"] Nov 03 16:43:33.394242 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:33.394216 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-hwn7b"] Nov 03 16:43:34.565882 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:34.565849 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9158c889-3243-4483-a481-3b41f97f0350" path="/var/lib/kubelet/pods/9158c889-3243-4483-a481-3b41f97f0350/volumes" Nov 03 16:43:36.086734 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:36.084954 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8" Nov 03 16:43:36.086734 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:36.086332 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m" Nov 03 16:43:41.465762 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:41.465728 4947 generic.go:356] "Generic (PLEG): container finished" podID="0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" containerID="5dba12412ca78a005f1d5e2cb6829c1d43d13f2f0af966da3d7860b6b2e17266" exitCode=0 Nov 03 16:43:41.466224 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:41.465809 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" event={"ID":"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3","Type":"ContainerDied","Data":"5dba12412ca78a005f1d5e2cb6829c1d43d13f2f0af966da3d7860b6b2e17266"} Nov 03 16:43:41.466277 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:41.466239 4947 scope.go:122] "RemoveContainer" containerID="5dba12412ca78a005f1d5e2cb6829c1d43d13f2f0af966da3d7860b6b2e17266" Nov 03 16:43:41.467638 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:41.467604 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:43:41.467742 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:41.467675 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:43:41.467742 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:41.467734 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:43:42.475255 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:42.475214 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" event={"ID":"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3","Type":"ContainerStarted","Data":"f218af63a4c82a7ed8a633bf9979802cb2ad68eb90403ddcabe210bde9adda55"} Nov 03 16:43:42.691782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:42.691747 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:42.691994 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:42.691970 4947 patch_prober.go:29] interesting pod/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 16:43:42.692039 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:42.692021 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" podUID="0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 16:43:43.224411 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:43.224372 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6"] Nov 03 16:43:43.253316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:43.253286 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d"] Nov 03 16:43:43.691709 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:43.691673 4947 patch_prober.go:29] interesting pod/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 16:43:43.692118 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:43.691746 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" podUID="0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 16:43:44.692178 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:44.692147 4947 patch_prober.go:29] interesting pod/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 16:43:44.692587 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:44.692199 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" podUID="0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 16:43:45.382639 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.382602 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x"] Nov 03 16:43:45.383326 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.383310 4947 cpu_manager.go:414] "containerMap: removing container" podUID="9158c889-3243-4483-a481-3b41f97f0350" containerName="payload-processing" Nov 03 16:43:45.383374 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.383328 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9158c889-3243-4483-a481-3b41f97f0350" containerName="payload-processing" Nov 03 16:43:45.383433 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.383425 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9158c889-3243-4483-a481-3b41f97f0350" containerName="payload-processing" Nov 03 16:43:45.397261 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.397224 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x" Nov 03 16:43:45.454811 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.454309 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8650a6e8\"" Nov 03 16:43:45.455030 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.455002 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8650a6e8-dockercfg-2fxpq\"" Nov 03 16:43:45.469894 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.467342 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q67jv\" (UniqueName: \"kubernetes.io/projected/d4b24b1f-3cbd-435d-b59d-c3214a86c39e-kube-api-access-q67jv\") pod \"payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x\" (UID: \"d4b24b1f-3cbd-435d-b59d-c3214a86c39e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x" Nov 03 16:43:45.479841 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.479803 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4b24b1f-3cbd-435d-b59d-c3214a86c39e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x\" (UID: \"d4b24b1f-3cbd-435d-b59d-c3214a86c39e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x" Nov 03 16:43:45.526043 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.526003 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x"] Nov 03 16:43:45.605548 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.605505 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q67jv\" (UniqueName: \"kubernetes.io/projected/d4b24b1f-3cbd-435d-b59d-c3214a86c39e-kube-api-access-q67jv\") pod \"payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x\" (UID: \"d4b24b1f-3cbd-435d-b59d-c3214a86c39e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x" Nov 03 16:43:45.605955 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.605933 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4b24b1f-3cbd-435d-b59d-c3214a86c39e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x\" (UID: \"d4b24b1f-3cbd-435d-b59d-c3214a86c39e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x" Nov 03 16:43:45.673776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.661331 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4b24b1f-3cbd-435d-b59d-c3214a86c39e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x\" (UID: \"d4b24b1f-3cbd-435d-b59d-c3214a86c39e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x" Nov 03 16:43:45.716724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.716446 4947 patch_prober.go:29] interesting pod/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 16:43:45.716724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.716500 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" podUID="0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 16:43:45.799118 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.794736 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q67jv\" (UniqueName: \"kubernetes.io/projected/d4b24b1f-3cbd-435d-b59d-c3214a86c39e-kube-api-access-q67jv\") pod \"payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x\" (UID: \"d4b24b1f-3cbd-435d-b59d-c3214a86c39e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x" Nov 03 16:43:45.855735 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.853914 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h"] Nov 03 16:43:45.885776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.884474 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h" Nov 03 16:43:45.977725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:45.969968 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h"] Nov 03 16:43:46.041927 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:46.041300 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x" Nov 03 16:43:46.184566 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:46.076401 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46cx9\" (UniqueName: \"kubernetes.io/projected/3a41205a-a408-4718-8a07-eb1870d0e91f-kube-api-access-46cx9\") pod \"payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h\" (UID: \"3a41205a-a408-4718-8a07-eb1870d0e91f\") " pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h" Nov 03 16:43:46.184566 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:46.076561 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a41205a-a408-4718-8a07-eb1870d0e91f-plugins-config-volume\") pod \"payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h\" (UID: \"3a41205a-a408-4718-8a07-eb1870d0e91f\") " pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h" Nov 03 16:43:46.197808 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:46.190443 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a41205a-a408-4718-8a07-eb1870d0e91f-plugins-config-volume\") pod \"payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h\" (UID: \"3a41205a-a408-4718-8a07-eb1870d0e91f\") " pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h" Nov 03 16:43:46.197808 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:46.190727 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46cx9\" (UniqueName: \"kubernetes.io/projected/3a41205a-a408-4718-8a07-eb1870d0e91f-kube-api-access-46cx9\") pod \"payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h\" (UID: \"3a41205a-a408-4718-8a07-eb1870d0e91f\") " pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h" Nov 03 16:43:46.197808 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:46.192151 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a41205a-a408-4718-8a07-eb1870d0e91f-plugins-config-volume\") pod \"payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h\" (UID: \"3a41205a-a408-4718-8a07-eb1870d0e91f\") " pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h" Nov 03 16:43:46.586529 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:46.552944 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46cx9\" (UniqueName: \"kubernetes.io/projected/3a41205a-a408-4718-8a07-eb1870d0e91f-kube-api-access-46cx9\") pod \"payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h\" (UID: \"3a41205a-a408-4718-8a07-eb1870d0e91f\") " pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h" Nov 03 16:43:46.668237 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:46.668207 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h" Nov 03 16:43:46.693054 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:46.692250 4947 patch_prober.go:29] interesting pod/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 16:43:46.693054 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:46.692311 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" podUID="0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 16:43:46.943985 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:46.943949 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x"] Nov 03 16:43:46.944418 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:43:46.943983 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b24b1f_3cbd_435d_b59d_c3214a86c39e.slice/crio-c47ca6925fe8d008f28cb4d5a4b1dd82842f190f8bbf87993c3ee70ce6833eba WatchSource:0}: Error finding container c47ca6925fe8d008f28cb4d5a4b1dd82842f190f8bbf87993c3ee70ce6833eba: Status 404 returned error can't find the container with id c47ca6925fe8d008f28cb4d5a4b1dd82842f190f8bbf87993c3ee70ce6833eba Nov 03 16:43:47.040635 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:47.040597 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h"] Nov 03 16:43:47.714403 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:47.714363 4947 patch_prober.go:29] interesting pod/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Nov 03 16:43:47.714611 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:47.714438 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" podUID="0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Nov 03 16:43:47.715233 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:47.715203 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h" event={"ID":"3a41205a-a408-4718-8a07-eb1870d0e91f","Type":"ContainerStarted","Data":"abfb29ccdb43c0ee9813e885d1564d898f690df2d7978ff3b7599f230ba79899"} Nov 03 16:43:47.715461 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:47.715442 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h" Nov 03 16:43:47.715561 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:47.715548 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h" event={"ID":"3a41205a-a408-4718-8a07-eb1870d0e91f","Type":"ContainerStarted","Data":"af3c0552251b6868fbb2d275e9158b74adfb2e1d5dbad68a03ffb0e7a328640d"} Nov 03 16:43:47.717542 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:47.717423 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x" event={"ID":"d4b24b1f-3cbd-435d-b59d-c3214a86c39e","Type":"ContainerStarted","Data":"37dbb257c5bd1ce61ff9ac809eb6b78a66fd5dd01ad4221d89f6d0e825818609"} Nov 03 16:43:47.717542 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:47.717473 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x" event={"ID":"d4b24b1f-3cbd-435d-b59d-c3214a86c39e","Type":"ContainerStarted","Data":"c47ca6925fe8d008f28cb4d5a4b1dd82842f190f8bbf87993c3ee70ce6833eba"} Nov 03 16:43:47.717542 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:47.717499 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x" Nov 03 16:43:47.816809 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:47.816747 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x" podStartSLOduration=2.816731744 podStartE2EDuration="2.816731744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:47.815024098 +0000 UTC m=+1771.404024065" watchObservedRunningTime="2026-11-03 16:43:47.816731744 +0000 UTC m=+1771.405731711" Nov 03 16:43:47.818479 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:47.818447 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h" podStartSLOduration=2.818436238 podStartE2EDuration="2.818436238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:47.763146255 +0000 UTC m=+1771.352146222" watchObservedRunningTime="2026-11-03 16:43:47.818436238 +0000 UTC m=+1771.407436205" Nov 03 16:43:47.824017 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:47.823997 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:48.225099 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.225051 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6" podUID="f0781fe8-13dc-4f23-9574-9967398b0a91" containerName="payload-processing" containerID="cri-o://b0a74142c5f66592110f7d0cd75536871d9fb878e0ec39b7ee1a1b965e32692e" gracePeriod=25 Nov 03 16:43:48.254100 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.254059 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d" podUID="6560e01c-5031-49da-aea1-19d2df514d44" containerName="payload-pre-processing" containerID="cri-o://ce0a3d14ba35b8b1bc831fa8763234d9095cbada0f982749f25b8cb804ab2c91" gracePeriod=25 Nov 03 16:43:48.545494 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.545465 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6" Nov 03 16:43:48.588164 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.588142 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d" Nov 03 16:43:48.607828 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.605956 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0781fe8-13dc-4f23-9574-9967398b0a91-kube-api-access-bsw4p\" (UniqueName: \"kubernetes.io/projected/f0781fe8-13dc-4f23-9574-9967398b0a91-kube-api-access-bsw4p\") pod \"f0781fe8-13dc-4f23-9574-9967398b0a91\" (UID: \"f0781fe8-13dc-4f23-9574-9967398b0a91\") " Nov 03 16:43:48.607828 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.606317 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0781fe8-13dc-4f23-9574-9967398b0a91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0781fe8-13dc-4f23-9574-9967398b0a91-plugins-config-volume\") pod \"f0781fe8-13dc-4f23-9574-9967398b0a91\" (UID: \"f0781fe8-13dc-4f23-9574-9967398b0a91\") " Nov 03 16:43:48.607828 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.607227 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0781fe8-13dc-4f23-9574-9967398b0a91-plugins-config-volume" pod "f0781fe8-13dc-4f23-9574-9967398b0a91" (UID: "f0781fe8-13dc-4f23-9574-9967398b0a91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:43:48.610422 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.610395 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0781fe8-13dc-4f23-9574-9967398b0a91-kube-api-access-bsw4p" pod "f0781fe8-13dc-4f23-9574-9967398b0a91" (UID: "f0781fe8-13dc-4f23-9574-9967398b0a91"). InnerVolumeSpecName "kube-api-access-bsw4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:43:48.696317 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.696289 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:48.708176 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.708149 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6560e01c-5031-49da-aea1-19d2df514d44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6560e01c-5031-49da-aea1-19d2df514d44-plugins-config-volume\") pod \"6560e01c-5031-49da-aea1-19d2df514d44\" (UID: \"6560e01c-5031-49da-aea1-19d2df514d44\") " Nov 03 16:43:48.708338 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.708324 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6560e01c-5031-49da-aea1-19d2df514d44-kube-api-access-d6cmh\" (UniqueName: \"kubernetes.io/projected/6560e01c-5031-49da-aea1-19d2df514d44-kube-api-access-d6cmh\") pod \"6560e01c-5031-49da-aea1-19d2df514d44\" (UID: \"6560e01c-5031-49da-aea1-19d2df514d44\") " Nov 03 16:43:48.708555 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.708531 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6560e01c-5031-49da-aea1-19d2df514d44-plugins-config-volume" pod "6560e01c-5031-49da-aea1-19d2df514d44" (UID: "6560e01c-5031-49da-aea1-19d2df514d44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:43:48.708959 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.708927 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6560e01c-5031-49da-aea1-19d2df514d44-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:43:48.708959 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.708955 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0781fe8-13dc-4f23-9574-9967398b0a91-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:43:48.709064 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.708969 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsw4p\" (UniqueName: \"kubernetes.io/projected/f0781fe8-13dc-4f23-9574-9967398b0a91-kube-api-access-bsw4p\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:43:48.710584 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.710565 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6560e01c-5031-49da-aea1-19d2df514d44-kube-api-access-d6cmh" pod "6560e01c-5031-49da-aea1-19d2df514d44" (UID: "6560e01c-5031-49da-aea1-19d2df514d44"). InnerVolumeSpecName "kube-api-access-d6cmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:43:48.725051 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.725020 4947 generic.go:356] "Generic (PLEG): container finished" podID="6560e01c-5031-49da-aea1-19d2df514d44" containerID="ce0a3d14ba35b8b1bc831fa8763234d9095cbada0f982749f25b8cb804ab2c91" exitCode=0 Nov 03 16:43:48.725183 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.725078 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d" Nov 03 16:43:48.725183 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.725104 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d" event={"ID":"6560e01c-5031-49da-aea1-19d2df514d44","Type":"ContainerDied","Data":"ce0a3d14ba35b8b1bc831fa8763234d9095cbada0f982749f25b8cb804ab2c91"} Nov 03 16:43:48.725183 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.725161 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d" event={"ID":"6560e01c-5031-49da-aea1-19d2df514d44","Type":"ContainerDied","Data":"fe60a3b23a3d5093dc4e63e62760496fa9ec77eb66ac3dcc116d03710febccf5"} Nov 03 16:43:48.725346 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.725193 4947 scope.go:122] "RemoveContainer" containerID="ce0a3d14ba35b8b1bc831fa8763234d9095cbada0f982749f25b8cb804ab2c91" Nov 03 16:43:48.726823 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.726796 4947 generic.go:356] "Generic (PLEG): container finished" podID="f0781fe8-13dc-4f23-9574-9967398b0a91" containerID="b0a74142c5f66592110f7d0cd75536871d9fb878e0ec39b7ee1a1b965e32692e" exitCode=0 Nov 03 16:43:48.726952 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.726851 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6" Nov 03 16:43:48.726952 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.726900 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6" event={"ID":"f0781fe8-13dc-4f23-9574-9967398b0a91","Type":"ContainerDied","Data":"b0a74142c5f66592110f7d0cd75536871d9fb878e0ec39b7ee1a1b965e32692e"} Nov 03 16:43:48.726952 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.726946 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6" event={"ID":"f0781fe8-13dc-4f23-9574-9967398b0a91","Type":"ContainerDied","Data":"da952f5775eb00db822feb6a27460d447d9611d17a6810d8c18099555e1ed5e5"} Nov 03 16:43:48.728567 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.728550 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:43:48.739028 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.737436 4947 scope.go:122] "RemoveContainer" containerID="ce0a3d14ba35b8b1bc831fa8763234d9095cbada0f982749f25b8cb804ab2c91" Nov 03 16:43:48.739028 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:43:48.737891 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce0a3d14ba35b8b1bc831fa8763234d9095cbada0f982749f25b8cb804ab2c91\": container with ID starting with ce0a3d14ba35b8b1bc831fa8763234d9095cbada0f982749f25b8cb804ab2c91 not found: ID does not exist" containerID="ce0a3d14ba35b8b1bc831fa8763234d9095cbada0f982749f25b8cb804ab2c91" Nov 03 16:43:48.739028 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.737927 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0a3d14ba35b8b1bc831fa8763234d9095cbada0f982749f25b8cb804ab2c91"} err="failed to get container status \"ce0a3d14ba35b8b1bc831fa8763234d9095cbada0f982749f25b8cb804ab2c91\": rpc error: code = NotFound desc = could not find container \"ce0a3d14ba35b8b1bc831fa8763234d9095cbada0f982749f25b8cb804ab2c91\": container with ID starting with ce0a3d14ba35b8b1bc831fa8763234d9095cbada0f982749f25b8cb804ab2c91 not found: ID does not exist" Nov 03 16:43:48.739028 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.737954 4947 scope.go:122] "RemoveContainer" containerID="b0a74142c5f66592110f7d0cd75536871d9fb878e0ec39b7ee1a1b965e32692e" Nov 03 16:43:48.747891 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.747849 4947 scope.go:122] "RemoveContainer" containerID="b0a74142c5f66592110f7d0cd75536871d9fb878e0ec39b7ee1a1b965e32692e" Nov 03 16:43:48.748195 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:43:48.748170 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0a74142c5f66592110f7d0cd75536871d9fb878e0ec39b7ee1a1b965e32692e\": container with ID starting with b0a74142c5f66592110f7d0cd75536871d9fb878e0ec39b7ee1a1b965e32692e not found: ID does not exist" containerID="b0a74142c5f66592110f7d0cd75536871d9fb878e0ec39b7ee1a1b965e32692e" Nov 03 16:43:48.748262 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.748201 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0a74142c5f66592110f7d0cd75536871d9fb878e0ec39b7ee1a1b965e32692e"} err="failed to get container status \"b0a74142c5f66592110f7d0cd75536871d9fb878e0ec39b7ee1a1b965e32692e\": rpc error: code = NotFound desc = could not find container \"b0a74142c5f66592110f7d0cd75536871d9fb878e0ec39b7ee1a1b965e32692e\": container with ID starting with b0a74142c5f66592110f7d0cd75536871d9fb878e0ec39b7ee1a1b965e32692e not found: ID does not exist" Nov 03 16:43:48.797540 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.797509 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6"] Nov 03 16:43:48.803102 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.803071 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-6fa98e-648c6fd65-fkbs6"] Nov 03 16:43:48.810609 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.810582 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6cmh\" (UniqueName: \"kubernetes.io/projected/6560e01c-5031-49da-aea1-19d2df514d44-kube-api-access-d6cmh\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:43:48.898930 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.898897 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d"] Nov 03 16:43:48.906339 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:48.906311 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-6fa98e-5c9cbd6cf9-v629d"] Nov 03 16:43:50.565998 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:50.565968 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6560e01c-5031-49da-aea1-19d2df514d44" path="/var/lib/kubelet/pods/6560e01c-5031-49da-aea1-19d2df514d44/volumes" Nov 03 16:43:50.566416 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:50.566401 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0781fe8-13dc-4f23-9574-9967398b0a91" path="/var/lib/kubelet/pods/f0781fe8-13dc-4f23-9574-9967398b0a91/volumes" Nov 03 16:43:51.814013 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:51.813982 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx"] Nov 03 16:43:51.814633 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:51.814620 4947 cpu_manager.go:414] "containerMap: removing container" podUID="6560e01c-5031-49da-aea1-19d2df514d44" containerName="payload-pre-processing" Nov 03 16:43:51.814633 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:51.814633 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6560e01c-5031-49da-aea1-19d2df514d44" containerName="payload-pre-processing" Nov 03 16:43:51.814758 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:51.814669 4947 cpu_manager.go:414] "containerMap: removing container" podUID="f0781fe8-13dc-4f23-9574-9967398b0a91" containerName="payload-processing" Nov 03 16:43:51.814758 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:51.814677 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0781fe8-13dc-4f23-9574-9967398b0a91" containerName="payload-processing" Nov 03 16:43:51.814840 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:51.814828 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6560e01c-5031-49da-aea1-19d2df514d44" containerName="payload-pre-processing" Nov 03 16:43:51.814879 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:51.814849 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0781fe8-13dc-4f23-9574-9967398b0a91" containerName="payload-processing" Nov 03 16:43:51.818255 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:51.818240 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx" Nov 03 16:43:51.847571 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:51.847545 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-156c2a-dockercfg-4t2gf\"" Nov 03 16:43:51.853611 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:51.853579 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx"] Nov 03 16:43:51.857892 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:51.857857 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-156c2a\"" Nov 03 16:43:51.875685 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:51.869274 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m9lx\" (UniqueName: \"kubernetes.io/projected/fcbcf982-571a-4281-ac51-d6c5fb9592bc-kube-api-access-5m9lx\") pod \"payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx\" (UID: \"fcbcf982-571a-4281-ac51-d6c5fb9592bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx" Nov 03 16:43:51.875685 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:51.869757 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcbcf982-571a-4281-ac51-d6c5fb9592bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx\" (UID: \"fcbcf982-571a-4281-ac51-d6c5fb9592bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx" Nov 03 16:43:51.963721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:51.963225 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt"] Nov 03 16:43:51.988722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:51.985977 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m9lx\" (UniqueName: \"kubernetes.io/projected/fcbcf982-571a-4281-ac51-d6c5fb9592bc-kube-api-access-5m9lx\") pod \"payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx\" (UID: \"fcbcf982-571a-4281-ac51-d6c5fb9592bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx" Nov 03 16:43:52.067777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:52.061248 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcbcf982-571a-4281-ac51-d6c5fb9592bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx\" (UID: \"fcbcf982-571a-4281-ac51-d6c5fb9592bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx" Nov 03 16:43:52.127783 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:52.126014 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcbcf982-571a-4281-ac51-d6c5fb9592bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx\" (UID: \"fcbcf982-571a-4281-ac51-d6c5fb9592bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx" Nov 03 16:43:52.145649 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:52.145615 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt"] Nov 03 16:43:52.145846 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:52.145767 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt" Nov 03 16:43:52.146589 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:52.146560 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m9lx\" (UniqueName: \"kubernetes.io/projected/fcbcf982-571a-4281-ac51-d6c5fb9592bc-kube-api-access-5m9lx\") pod \"payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx\" (UID: \"fcbcf982-571a-4281-ac51-d6c5fb9592bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx" Nov 03 16:43:52.266860 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:52.266825 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8t6s\" (UniqueName: \"kubernetes.io/projected/b072b11d-bfb0-432e-b5d6-65d887f58cc4-kube-api-access-q8t6s\") pod \"payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt\" (UID: \"b072b11d-bfb0-432e-b5d6-65d887f58cc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt" Nov 03 16:43:52.267043 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:52.266972 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b072b11d-bfb0-432e-b5d6-65d887f58cc4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt\" (UID: \"b072b11d-bfb0-432e-b5d6-65d887f58cc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt" Nov 03 16:43:52.368282 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:52.368202 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8t6s\" (UniqueName: \"kubernetes.io/projected/b072b11d-bfb0-432e-b5d6-65d887f58cc4-kube-api-access-q8t6s\") pod \"payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt\" (UID: \"b072b11d-bfb0-432e-b5d6-65d887f58cc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt" Nov 03 16:43:52.368423 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:52.368302 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b072b11d-bfb0-432e-b5d6-65d887f58cc4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt\" (UID: \"b072b11d-bfb0-432e-b5d6-65d887f58cc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt" Nov 03 16:43:52.368979 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:52.368960 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b072b11d-bfb0-432e-b5d6-65d887f58cc4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt\" (UID: \"b072b11d-bfb0-432e-b5d6-65d887f58cc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt" Nov 03 16:43:52.393229 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:52.393202 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8t6s\" (UniqueName: \"kubernetes.io/projected/b072b11d-bfb0-432e-b5d6-65d887f58cc4-kube-api-access-q8t6s\") pod \"payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt\" (UID: \"b072b11d-bfb0-432e-b5d6-65d887f58cc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt" Nov 03 16:43:52.430946 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:52.430914 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx" Nov 03 16:43:52.508745 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:52.508351 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt" Nov 03 16:43:52.736859 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:43:52.736832 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcbcf982_571a_4281_ac51_d6c5fb9592bc.slice/crio-baac0d845c6a5e0cc6db605f43b6adb0234baaedbc8efafe771f6a3a414cc94d WatchSource:0}: Error finding container baac0d845c6a5e0cc6db605f43b6adb0234baaedbc8efafe771f6a3a414cc94d: Status 404 returned error can't find the container with id baac0d845c6a5e0cc6db605f43b6adb0234baaedbc8efafe771f6a3a414cc94d Nov 03 16:43:52.738960 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:52.738937 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx"] Nov 03 16:43:52.759755 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:52.759691 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx" event={"ID":"fcbcf982-571a-4281-ac51-d6c5fb9592bc","Type":"ContainerStarted","Data":"baac0d845c6a5e0cc6db605f43b6adb0234baaedbc8efafe771f6a3a414cc94d"} Nov 03 16:43:52.847212 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:43:52.847175 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb072b11d_bfb0_432e_b5d6_65d887f58cc4.slice/crio-19c6753dcf97f3d842f37c553f9c84c3248ff1083c8fd30979d91a27b6b0c230 WatchSource:0}: Error finding container 19c6753dcf97f3d842f37c553f9c84c3248ff1083c8fd30979d91a27b6b0c230: Status 404 returned error can't find the container with id 19c6753dcf97f3d842f37c553f9c84c3248ff1083c8fd30979d91a27b6b0c230 Nov 03 16:43:52.847540 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:52.847304 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt"] Nov 03 16:43:52.934234 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:52.934184 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:43:53.823876 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:53.822080 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx" event={"ID":"fcbcf982-571a-4281-ac51-d6c5fb9592bc","Type":"ContainerStarted","Data":"62426d31bdc6aec14760cb1a9119610833e864250ca285ee85e249390fa2f463"} Nov 03 16:43:53.823876 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:53.823555 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx" Nov 03 16:43:53.842056 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:53.842010 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt" event={"ID":"b072b11d-bfb0-432e-b5d6-65d887f58cc4","Type":"ContainerStarted","Data":"2644b54ba59bde1b31e58ff306b57ffb936917933d9138457ddaac2d671bfdfa"} Nov 03 16:43:53.842401 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:53.842366 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt" event={"ID":"b072b11d-bfb0-432e-b5d6-65d887f58cc4","Type":"ContainerStarted","Data":"19c6753dcf97f3d842f37c553f9c84c3248ff1083c8fd30979d91a27b6b0c230"} Nov 03 16:43:53.852846 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:53.843502 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt" Nov 03 16:43:53.879349 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:53.879278 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx" podStartSLOduration=2.879256041 podStartE2EDuration="2.879256041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:53.878065459 +0000 UTC m=+1777.467065428" watchObservedRunningTime="2026-11-03 16:43:53.879256041 +0000 UTC m=+1777.468256012" Nov 03 16:43:54.836095 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:54.836027 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt" podStartSLOduration=3.836011478 podStartE2EDuration="3.836011478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:53.930175814 +0000 UTC m=+1777.519175782" watchObservedRunningTime="2026-11-03 16:43:54.836011478 +0000 UTC m=+1778.425011448" Nov 03 16:43:54.837036 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:54.837012 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr"] Nov 03 16:43:54.840435 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:54.840411 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:43:54.852611 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:54.852581 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f7c9da9-3f09-42c2-8b73-3a704acf074d-maas-api-tls\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr\" (UID: \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:43:54.852905 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:54.852873 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc29g\" (UniqueName: \"kubernetes.io/projected/6f7c9da9-3f09-42c2-8b73-3a704acf074d-kube-api-access-rc29g\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr\" (UID: \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:43:54.853321 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:54.852969 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f7c9da9-3f09-42c2-8b73-3a704acf074d-openshift-service-ca\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr\" (UID: \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:43:54.853321 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:54.853022 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f7c9da9-3f09-42c2-8b73-3a704acf074d-metrics-certs\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr\" (UID: \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:43:54.856272 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:54.856249 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8650a6e8-serving-cert\"" Nov 03 16:43:54.876438 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:54.876269 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr"] Nov 03 16:43:54.954318 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:54.954289 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f7c9da9-3f09-42c2-8b73-3a704acf074d-maas-api-tls\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr\" (UID: \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:43:54.954493 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:54.954394 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc29g\" (UniqueName: \"kubernetes.io/projected/6f7c9da9-3f09-42c2-8b73-3a704acf074d-kube-api-access-rc29g\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr\" (UID: \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:43:54.954493 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:43:54.954434 4947 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-serving-cert: secret "maas-api-e2e-mt-8650a6e8-serving-cert" not found Nov 03 16:43:54.954493 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:54.954455 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f7c9da9-3f09-42c2-8b73-3a704acf074d-openshift-service-ca\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr\" (UID: \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:43:54.954598 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:43:54.954493 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f7c9da9-3f09-42c2-8b73-3a704acf074d-maas-api-tls podName:6f7c9da9-3f09-42c2-8b73-3a704acf074d nodeName:}" failed. No retries permitted until 2026-11-03 16:43:55.454477847 +0000 UTC m=+1779.043477812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f7c9da9-3f09-42c2-8b73-3a704acf074d-maas-api-tls") pod "maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" (UID: "6f7c9da9-3f09-42c2-8b73-3a704acf074d") : secret "maas-api-e2e-mt-8650a6e8-serving-cert" not found Nov 03 16:43:54.954598 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:54.954511 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f7c9da9-3f09-42c2-8b73-3a704acf074d-metrics-certs\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr\" (UID: \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:43:54.955174 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:54.955139 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f7c9da9-3f09-42c2-8b73-3a704acf074d-openshift-service-ca\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr\" (UID: \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:43:54.957189 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:54.957166 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f7c9da9-3f09-42c2-8b73-3a704acf074d-metrics-certs\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr\" (UID: \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:43:54.981420 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:54.981387 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc29g\" (UniqueName: \"kubernetes.io/projected/6f7c9da9-3f09-42c2-8b73-3a704acf074d-kube-api-access-rc29g\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr\" (UID: \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:43:55.481350 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:55.481295 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f7c9da9-3f09-42c2-8b73-3a704acf074d-maas-api-tls\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr\" (UID: \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:43:55.483938 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:55.483916 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f7c9da9-3f09-42c2-8b73-3a704acf074d-maas-api-tls\") pod \"maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr\" (UID: \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:43:55.755194 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:55.755162 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:43:56.209432 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:56.202577 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr"] Nov 03 16:43:56.257412 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:56.256616 4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:43:56.882315 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:56.882279 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" event={"ID":"6f7c9da9-3f09-42c2-8b73-3a704acf074d","Type":"ContainerStarted","Data":"67114341aca30641363b5c80e59c33f0c69ebc555499d79efce4ae37825aef84"} Nov 03 16:43:56.882509 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:56.882329 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" event={"ID":"6f7c9da9-3f09-42c2-8b73-3a704acf074d","Type":"ContainerStarted","Data":"2b651e1fa3a8f647be3b58c6097ce65042a5bcf8efb81900e1a11aef22e9d353"} Nov 03 16:43:56.882509 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:56.882411 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:43:56.936770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:56.936662 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" podStartSLOduration=2.500137446 podStartE2EDuration="2.936645997s" totalImagesPullingTime="436.508551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:43:56.934846824 +0000 UTC m=+1780.523846791" watchObservedRunningTime="2026-11-03 16:43:56.936645997 +0000 UTC m=+1780.525646013" Nov 03 16:43:58.757735 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:58.756910 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h" Nov 03 16:43:58.759928 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:43:58.759896 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x" Nov 03 16:44:00.131293 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:44:00.131253 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:44:02.989589 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:02.989551 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:44:05.878049 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:05.878018 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt" Nov 03 16:44:05.878491 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:05.878099 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx" Nov 03 16:44:06.013678 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:06.013647 4947 patch_prober.go:29] interesting pod/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 16:44:06.013859 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:06.013723 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" podUID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 16:44:06.111012 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:06.110979 4947 generic.go:356] "Generic (PLEG): container finished" podID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" containerID="9942e727094dd7306b774fa7a1b12ae4965247ec53b3428af1234165a67c53b2" exitCode=0 Nov 03 16:44:06.111201 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:06.111053 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" event={"ID":"595b44f5-49e0-4cd7-bd8f-83a72f6f426c","Type":"ContainerDied","Data":"9942e727094dd7306b774fa7a1b12ae4965247ec53b3428af1234165a67c53b2"} Nov 03 16:44:06.111546 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:06.111529 4947 scope.go:122] "RemoveContainer" containerID="9942e727094dd7306b774fa7a1b12ae4965247ec53b3428af1234165a67c53b2" Nov 03 16:44:06.113116 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:06.113087 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:44:06.113166 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:06.113155 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:44:06.113204 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:06.113185 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:44:07.047069 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.047033 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr"] Nov 03 16:44:07.050506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.050486 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" Nov 03 16:44:07.077854 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.077824 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-156c2a-serving-cert\"" Nov 03 16:44:07.082879 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.082844 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr"] Nov 03 16:44:07.121463 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.120655 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" event={"ID":"595b44f5-49e0-4cd7-bd8f-83a72f6f426c","Type":"ContainerStarted","Data":"60f5c148d442bd24a3b74e4ac53374f25160c1a1c2027c53d4383ebce6d9a577"} Nov 03 16:44:07.178828 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.178792 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d73ebc6-5b5f-4d67-af5c-f424bf672754-maas-api-tls\") pod \"maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr\" (UID: \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" Nov 03 16:44:07.179021 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.179000 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hndnm\" (UniqueName: \"kubernetes.io/projected/0d73ebc6-5b5f-4d67-af5c-f424bf672754-kube-api-access-hndnm\") pod \"maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr\" (UID: \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" Nov 03 16:44:07.179075 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.179056 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d73ebc6-5b5f-4d67-af5c-f424bf672754-metrics-certs\") pod \"maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr\" (UID: \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" Nov 03 16:44:07.179205 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.179189 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d73ebc6-5b5f-4d67-af5c-f424bf672754-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr\" (UID: \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" Nov 03 16:44:07.221599 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.221566 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:44:07.221821 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.221804 4947 patch_prober.go:29] interesting pod/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 16:44:07.221880 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.221845 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" podUID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 16:44:07.283720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.281453 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d73ebc6-5b5f-4d67-af5c-f424bf672754-maas-api-tls\") pod \"maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr\" (UID: \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" Nov 03 16:44:07.283720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.281879 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hndnm\" (UniqueName: \"kubernetes.io/projected/0d73ebc6-5b5f-4d67-af5c-f424bf672754-kube-api-access-hndnm\") pod \"maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr\" (UID: \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" Nov 03 16:44:07.283720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.281947 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d73ebc6-5b5f-4d67-af5c-f424bf672754-metrics-certs\") pod \"maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr\" (UID: \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" Nov 03 16:44:07.283720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.282114 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d73ebc6-5b5f-4d67-af5c-f424bf672754-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr\" (UID: \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" Nov 03 16:44:07.283720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.283003 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d73ebc6-5b5f-4d67-af5c-f424bf672754-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr\" (UID: \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" Nov 03 16:44:07.321764 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.321145 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d73ebc6-5b5f-4d67-af5c-f424bf672754-metrics-certs\") pod \"maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr\" (UID: \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" Nov 03 16:44:07.346638 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.346604 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d73ebc6-5b5f-4d67-af5c-f424bf672754-maas-api-tls\") pod \"maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr\" (UID: \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" Nov 03 16:44:07.365405 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.365372 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hndnm\" (UniqueName: \"kubernetes.io/projected/0d73ebc6-5b5f-4d67-af5c-f424bf672754-kube-api-access-hndnm\") pod \"maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr\" (UID: \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" Nov 03 16:44:07.664957 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:07.664873 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" Nov 03 16:44:08.037549 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:08.036924 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr"] Nov 03 16:44:08.064723 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:44:08.061113 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d73ebc6_5b5f_4d67_af5c_f424bf672754.slice/crio-bfe0465929c8004cf53847ee7aa2971cf5189e78887b3e57fb7d336944795792 WatchSource:0}: Error finding container bfe0465929c8004cf53847ee7aa2971cf5189e78887b3e57fb7d336944795792: Status 404 returned error can't find the container with id bfe0465929c8004cf53847ee7aa2971cf5189e78887b3e57fb7d336944795792 Nov 03 16:44:08.140837 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:08.140805 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" event={"ID":"0d73ebc6-5b5f-4d67-af5c-f424bf672754","Type":"ContainerStarted","Data":"bfe0465929c8004cf53847ee7aa2971cf5189e78887b3e57fb7d336944795792"} Nov 03 16:44:08.235220 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:08.229197 4947 patch_prober.go:29] interesting pod/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 16:44:08.235220 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:08.229282 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" podUID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 16:44:09.367785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:09.271232 4947 patch_prober.go:29] interesting pod/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 16:44:09.367785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:09.271297 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" podUID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 16:44:09.485145 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:09.485103 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" event={"ID":"0d73ebc6-5b5f-4d67-af5c-f424bf672754","Type":"ContainerStarted","Data":"afce7dcc7c22cf23a7da04792d4f7fde5653d888b123ca3a93f04442364ea081"} Nov 03 16:44:09.487009 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:09.486493 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" Nov 03 16:44:10.255789 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:10.255678 4947 patch_prober.go:29] interesting pod/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 16:44:10.256085 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:10.256054 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" podUID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 16:44:11.223116 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:11.223080 4947 patch_prober.go:29] interesting pod/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 16:44:11.223803 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:11.223760 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" podUID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 16:44:12.223206 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:12.223165 4947 patch_prober.go:29] interesting pod/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 16:44:12.223819 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:12.223223 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" podUID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 16:44:13.256778 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:13.253060 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 16:44:13.256778 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:13.254784 4947 patch_prober.go:29] interesting pod/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 16:44:13.256778 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:13.254826 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" podUID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 16:44:13.283721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:13.278777 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:44:13.626796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:13.620168 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 16:44:14.249721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:14.248169 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:44:14.249721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:14.249416 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:44:14.260212 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:14.259924 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:44:14.337214 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:14.337164 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" podStartSLOduration=6.761541623 podStartE2EDuration="7.337151768s" totalImagesPullingTime="575.610145ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:09.550084566 +0000 UTC m=+1793.139084537" watchObservedRunningTime="2026-11-03 16:44:14.337151768 +0000 UTC m=+1797.926151735" Nov 03 16:44:16.540577 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:16.540546 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" Nov 03 16:44:16.793874 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:16.793802 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:44:16.794262 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:16.794247 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:44:17.117571 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:17.117488 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw"] Nov 03 16:44:17.314432 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:17.314396 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt"] Nov 03 16:44:22.118601 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.118546 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw" podUID="c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf" containerName="payload-processing" containerID="cri-o://1c6609f82934dcd9d23145201785f2a22e5a9ec7dc2159eb59468028d55debfc" gracePeriod=25 Nov 03 16:44:22.315523 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.315303 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt" podUID="4bed0898-f1a8-47ed-b47f-f6df2b263d74" containerName="payload-pre-processing" containerID="cri-o://5406fa4dc12ea35b734b4f950169f95b09c59eb7b1f674afb301e962dcf61aa1" gracePeriod=25 Nov 03 16:44:22.456215 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.456185 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw" Nov 03 16:44:22.523300 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.523262 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf-kube-api-access-ckbll\" (UniqueName: \"kubernetes.io/projected/c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf-kube-api-access-ckbll\") pod \"c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf\" (UID: \"c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf\") " Nov 03 16:44:22.523493 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.523382 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf-plugins-config-volume\") pod \"c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf\" (UID: \"c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf\") " Nov 03 16:44:22.525137 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.524745 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf-plugins-config-volume" pod "c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf" (UID: "c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:44:22.528251 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.528224 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf-kube-api-access-ckbll" pod "c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf" (UID: "c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf"). InnerVolumeSpecName "kube-api-access-ckbll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:22.588834 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.588812 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt" Nov 03 16:44:22.625340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.625308 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckbll\" (UniqueName: \"kubernetes.io/projected/c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf-kube-api-access-ckbll\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:22.625340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.625340 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:22.726026 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.725948 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bed0898-f1a8-47ed-b47f-f6df2b263d74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bed0898-f1a8-47ed-b47f-f6df2b263d74-plugins-config-volume\") pod \"4bed0898-f1a8-47ed-b47f-f6df2b263d74\" (UID: \"4bed0898-f1a8-47ed-b47f-f6df2b263d74\") " Nov 03 16:44:22.726189 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.726097 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bed0898-f1a8-47ed-b47f-f6df2b263d74-kube-api-access-mgwr7\" (UniqueName: \"kubernetes.io/projected/4bed0898-f1a8-47ed-b47f-f6df2b263d74-kube-api-access-mgwr7\") pod \"4bed0898-f1a8-47ed-b47f-f6df2b263d74\" (UID: \"4bed0898-f1a8-47ed-b47f-f6df2b263d74\") " Nov 03 16:44:22.726384 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.726359 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bed0898-f1a8-47ed-b47f-f6df2b263d74-plugins-config-volume" pod "4bed0898-f1a8-47ed-b47f-f6df2b263d74" (UID: "4bed0898-f1a8-47ed-b47f-f6df2b263d74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:44:22.726737 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.726722 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bed0898-f1a8-47ed-b47f-f6df2b263d74-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:22.728286 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.728266 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bed0898-f1a8-47ed-b47f-f6df2b263d74-kube-api-access-mgwr7" pod "4bed0898-f1a8-47ed-b47f-f6df2b263d74" (UID: "4bed0898-f1a8-47ed-b47f-f6df2b263d74"). InnerVolumeSpecName "kube-api-access-mgwr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:22.827673 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.827631 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgwr7\" (UniqueName: \"kubernetes.io/projected/4bed0898-f1a8-47ed-b47f-f6df2b263d74-kube-api-access-mgwr7\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:22.830601 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.830573 4947 generic.go:356] "Generic (PLEG): container finished" podID="c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf" containerID="1c6609f82934dcd9d23145201785f2a22e5a9ec7dc2159eb59468028d55debfc" exitCode=0 Nov 03 16:44:22.830754 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.830615 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw" Nov 03 16:44:22.830754 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.830651 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw" event={"ID":"c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf","Type":"ContainerDied","Data":"1c6609f82934dcd9d23145201785f2a22e5a9ec7dc2159eb59468028d55debfc"} Nov 03 16:44:22.830754 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.830716 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw" event={"ID":"c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf","Type":"ContainerDied","Data":"e60b4c19ecf34122e8d9ab06c8118202710dec948ad79d4267e9a06f62964840"} Nov 03 16:44:22.830754 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.830746 4947 scope.go:122] "RemoveContainer" containerID="1c6609f82934dcd9d23145201785f2a22e5a9ec7dc2159eb59468028d55debfc" Nov 03 16:44:22.832348 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.832325 4947 generic.go:356] "Generic (PLEG): container finished" podID="4bed0898-f1a8-47ed-b47f-f6df2b263d74" containerID="5406fa4dc12ea35b734b4f950169f95b09c59eb7b1f674afb301e962dcf61aa1" exitCode=0 Nov 03 16:44:22.832457 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.832375 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt" Nov 03 16:44:22.832457 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.832406 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt" event={"ID":"4bed0898-f1a8-47ed-b47f-f6df2b263d74","Type":"ContainerDied","Data":"5406fa4dc12ea35b734b4f950169f95b09c59eb7b1f674afb301e962dcf61aa1"} Nov 03 16:44:22.832457 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.832442 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt" event={"ID":"4bed0898-f1a8-47ed-b47f-f6df2b263d74","Type":"ContainerDied","Data":"c8c09832e27fb9aae0f3abf085efcc0486be509320b09853d0fdef20641da4fb"} Nov 03 16:44:22.854065 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.854038 4947 scope.go:122] "RemoveContainer" containerID="1c6609f82934dcd9d23145201785f2a22e5a9ec7dc2159eb59468028d55debfc" Nov 03 16:44:22.854482 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:44:22.854464 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6609f82934dcd9d23145201785f2a22e5a9ec7dc2159eb59468028d55debfc\": container with ID starting with 1c6609f82934dcd9d23145201785f2a22e5a9ec7dc2159eb59468028d55debfc not found: ID does not exist" containerID="1c6609f82934dcd9d23145201785f2a22e5a9ec7dc2159eb59468028d55debfc" Nov 03 16:44:22.854552 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.854493 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6609f82934dcd9d23145201785f2a22e5a9ec7dc2159eb59468028d55debfc"} err="failed to get container status \"1c6609f82934dcd9d23145201785f2a22e5a9ec7dc2159eb59468028d55debfc\": rpc error: code = NotFound desc = could not find container \"1c6609f82934dcd9d23145201785f2a22e5a9ec7dc2159eb59468028d55debfc\": container with ID starting with 1c6609f82934dcd9d23145201785f2a22e5a9ec7dc2159eb59468028d55debfc not found: ID does not exist" Nov 03 16:44:22.854552 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.854511 4947 scope.go:122] "RemoveContainer" containerID="5406fa4dc12ea35b734b4f950169f95b09c59eb7b1f674afb301e962dcf61aa1" Nov 03 16:44:22.864440 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.864422 4947 scope.go:122] "RemoveContainer" containerID="5406fa4dc12ea35b734b4f950169f95b09c59eb7b1f674afb301e962dcf61aa1" Nov 03 16:44:22.864754 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:44:22.864734 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5406fa4dc12ea35b734b4f950169f95b09c59eb7b1f674afb301e962dcf61aa1\": container with ID starting with 5406fa4dc12ea35b734b4f950169f95b09c59eb7b1f674afb301e962dcf61aa1 not found: ID does not exist" containerID="5406fa4dc12ea35b734b4f950169f95b09c59eb7b1f674afb301e962dcf61aa1" Nov 03 16:44:22.864806 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.864764 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5406fa4dc12ea35b734b4f950169f95b09c59eb7b1f674afb301e962dcf61aa1"} err="failed to get container status \"5406fa4dc12ea35b734b4f950169f95b09c59eb7b1f674afb301e962dcf61aa1\": rpc error: code = NotFound desc = could not find container \"5406fa4dc12ea35b734b4f950169f95b09c59eb7b1f674afb301e962dcf61aa1\": container with ID starting with 5406fa4dc12ea35b734b4f950169f95b09c59eb7b1f674afb301e962dcf61aa1 not found: ID does not exist" Nov 03 16:44:22.910441 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.910408 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw"] Nov 03 16:44:22.914756 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.914721 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4e3be7-79584f8f57-d92dw"] Nov 03 16:44:22.951411 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.951379 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt"] Nov 03 16:44:22.954953 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:22.954904 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4e3be7-648d758d7-b4sdt"] Nov 03 16:44:24.571033 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:24.571002 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bed0898-f1a8-47ed-b47f-f6df2b263d74" path="/var/lib/kubelet/pods/4bed0898-f1a8-47ed-b47f-f6df2b263d74/volumes" Nov 03 16:44:24.571437 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:24.571423 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf" path="/var/lib/kubelet/pods/c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf/volumes" Nov 03 16:44:25.680728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:25.665936 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:44:27.917120 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:27.917075 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4"] Nov 03 16:44:27.918679 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:27.918650 4947 cpu_manager.go:414] "containerMap: removing container" podUID="c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf" containerName="payload-processing" Nov 03 16:44:27.918679 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:27.918678 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf" containerName="payload-processing" Nov 03 16:44:27.918891 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:27.918724 4947 cpu_manager.go:414] "containerMap: removing container" podUID="4bed0898-f1a8-47ed-b47f-f6df2b263d74" containerName="payload-pre-processing" Nov 03 16:44:27.918891 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:27.918733 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bed0898-f1a8-47ed-b47f-f6df2b263d74" containerName="payload-pre-processing" Nov 03 16:44:27.919107 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:27.919089 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1979eb8-e4c8-4bd0-83b7-dd7cfd5f25cf" containerName="payload-processing" Nov 03 16:44:27.919158 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:27.919118 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bed0898-f1a8-47ed-b47f-f6df2b263d74" containerName="payload-pre-processing" Nov 03 16:44:27.926231 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:27.926205 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" Nov 03 16:44:27.945766 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:27.945675 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4"] Nov 03 16:44:27.951198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:27.950589 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-73cc79-serving-cert\"" Nov 03 16:44:27.985402 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:27.985364 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f1da65-4f9b-4512-b1fb-66c719d89abb-metrics-certs\") pod \"maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4\" (UID: \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" Nov 03 16:44:27.985597 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:27.985424 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f1da65-4f9b-4512-b1fb-66c719d89abb-maas-api-tls\") pod \"maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4\" (UID: \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" Nov 03 16:44:27.985597 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:27.985548 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f1da65-4f9b-4512-b1fb-66c719d89abb-openshift-service-ca\") pod \"maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4\" (UID: \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" Nov 03 16:44:27.985750 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:27.985724 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbs74\" (UniqueName: \"kubernetes.io/projected/e7f1da65-4f9b-4512-b1fb-66c719d89abb-kube-api-access-cbs74\") pod \"maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4\" (UID: \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" Nov 03 16:44:28.087010 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:28.086974 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f1da65-4f9b-4512-b1fb-66c719d89abb-metrics-certs\") pod \"maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4\" (UID: \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" Nov 03 16:44:28.087212 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:28.087164 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f1da65-4f9b-4512-b1fb-66c719d89abb-maas-api-tls\") pod \"maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4\" (UID: \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" Nov 03 16:44:28.087283 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:28.087208 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f1da65-4f9b-4512-b1fb-66c719d89abb-openshift-service-ca\") pod \"maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4\" (UID: \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" Nov 03 16:44:28.087365 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:28.087340 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbs74\" (UniqueName: \"kubernetes.io/projected/e7f1da65-4f9b-4512-b1fb-66c719d89abb-kube-api-access-cbs74\") pod \"maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4\" (UID: \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" Nov 03 16:44:28.088066 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:28.088003 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f1da65-4f9b-4512-b1fb-66c719d89abb-openshift-service-ca\") pod \"maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4\" (UID: \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" Nov 03 16:44:28.089853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:28.089831 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f1da65-4f9b-4512-b1fb-66c719d89abb-metrics-certs\") pod \"maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4\" (UID: \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" Nov 03 16:44:28.089937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:28.089885 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f1da65-4f9b-4512-b1fb-66c719d89abb-maas-api-tls\") pod \"maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4\" (UID: \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" Nov 03 16:44:28.112106 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:28.112080 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbs74\" (UniqueName: \"kubernetes.io/projected/e7f1da65-4f9b-4512-b1fb-66c719d89abb-kube-api-access-cbs74\") pod \"maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4\" (UID: \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" Nov 03 16:44:28.249728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:28.241956 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" Nov 03 16:44:28.635287 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:44:28.635255 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f1da65_4f9b_4512_b1fb_66c719d89abb.slice/crio-f2a43136a4459e9d415bbb278c9a07b0c73d63cfd9d7db82aa86ebec6a5fda7e WatchSource:0}: Error finding container f2a43136a4459e9d415bbb278c9a07b0c73d63cfd9d7db82aa86ebec6a5fda7e: Status 404 returned error can't find the container with id f2a43136a4459e9d415bbb278c9a07b0c73d63cfd9d7db82aa86ebec6a5fda7e Nov 03 16:44:28.637649 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:28.637620 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4"] Nov 03 16:44:28.939880 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:28.939736 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" event={"ID":"e7f1da65-4f9b-4512-b1fb-66c719d89abb","Type":"ContainerStarted","Data":"f2a43136a4459e9d415bbb278c9a07b0c73d63cfd9d7db82aa86ebec6a5fda7e"} Nov 03 16:44:29.959723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:29.959569 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" event={"ID":"e7f1da65-4f9b-4512-b1fb-66c719d89abb","Type":"ContainerStarted","Data":"21b8362638fd1ba2174ae72765c07f0db457264f4f6e37f5c4767ba9830e7210"} Nov 03 16:44:29.962755 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:29.960524 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" Nov 03 16:44:30.011050 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:30.010992 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" podStartSLOduration=2.628448877 podStartE2EDuration="3.01097502s" totalImagesPullingTime="382.526143ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:30.008878453 +0000 UTC m=+1813.597878439" watchObservedRunningTime="2026-11-03 16:44:30.01097502 +0000 UTC m=+1813.599974988" Nov 03 16:44:30.225792 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:44:30.223408 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:44:30.996219 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:30.996187 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2"] Nov 03 16:44:35.381815 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:35.381781 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt"] Nov 03 16:44:35.432943 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:35.432898 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx"] Nov 03 16:44:35.982357 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:35.982318 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" Nov 03 16:44:40.382322 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:40.382259 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt" podUID="b072b11d-bfb0-432e-b5d6-65d887f58cc4" containerName="payload-processing" containerID="cri-o://2644b54ba59bde1b31e58ff306b57ffb936917933d9138457ddaac2d671bfdfa" gracePeriod=25 Nov 03 16:44:40.433474 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:40.433402 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx" podUID="fcbcf982-571a-4281-ac51-d6c5fb9592bc" containerName="payload-pre-processing" containerID="cri-o://62426d31bdc6aec14760cb1a9119610833e864250ca285ee85e249390fa2f463" gracePeriod=25 Nov 03 16:44:40.735468 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:40.735427 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt" Nov 03 16:44:40.810172 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:40.810136 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx" Nov 03 16:44:40.864977 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:40.864936 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b072b11d-bfb0-432e-b5d6-65d887f58cc4-kube-api-access-q8t6s\" (UniqueName: \"kubernetes.io/projected/b072b11d-bfb0-432e-b5d6-65d887f58cc4-kube-api-access-q8t6s\") pod \"b072b11d-bfb0-432e-b5d6-65d887f58cc4\" (UID: \"b072b11d-bfb0-432e-b5d6-65d887f58cc4\") " Nov 03 16:44:40.865187 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:40.864997 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcbcf982-571a-4281-ac51-d6c5fb9592bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcbcf982-571a-4281-ac51-d6c5fb9592bc-plugins-config-volume\") pod \"fcbcf982-571a-4281-ac51-d6c5fb9592bc\" (UID: \"fcbcf982-571a-4281-ac51-d6c5fb9592bc\") " Nov 03 16:44:40.865187 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:40.865103 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcbcf982-571a-4281-ac51-d6c5fb9592bc-kube-api-access-5m9lx\" (UniqueName: \"kubernetes.io/projected/fcbcf982-571a-4281-ac51-d6c5fb9592bc-kube-api-access-5m9lx\") pod \"fcbcf982-571a-4281-ac51-d6c5fb9592bc\" (UID: \"fcbcf982-571a-4281-ac51-d6c5fb9592bc\") " Nov 03 16:44:40.865187 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:40.865147 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b072b11d-bfb0-432e-b5d6-65d887f58cc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b072b11d-bfb0-432e-b5d6-65d887f58cc4-plugins-config-volume\") pod \"b072b11d-bfb0-432e-b5d6-65d887f58cc4\" (UID: \"b072b11d-bfb0-432e-b5d6-65d887f58cc4\") " Nov 03 16:44:40.866243 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:40.866199 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcbcf982-571a-4281-ac51-d6c5fb9592bc-plugins-config-volume" pod "fcbcf982-571a-4281-ac51-d6c5fb9592bc" (UID: "fcbcf982-571a-4281-ac51-d6c5fb9592bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:44:40.866243 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:40.866198 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b072b11d-bfb0-432e-b5d6-65d887f58cc4-plugins-config-volume" pod "b072b11d-bfb0-432e-b5d6-65d887f58cc4" (UID: "b072b11d-bfb0-432e-b5d6-65d887f58cc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:44:40.868781 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:40.868741 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcbcf982-571a-4281-ac51-d6c5fb9592bc-kube-api-access-5m9lx" pod "fcbcf982-571a-4281-ac51-d6c5fb9592bc" (UID: "fcbcf982-571a-4281-ac51-d6c5fb9592bc"). InnerVolumeSpecName "kube-api-access-5m9lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:40.868971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:40.868946 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b072b11d-bfb0-432e-b5d6-65d887f58cc4-kube-api-access-q8t6s" pod "b072b11d-bfb0-432e-b5d6-65d887f58cc4" (UID: "b072b11d-bfb0-432e-b5d6-65d887f58cc4"). InnerVolumeSpecName "kube-api-access-q8t6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:40.967894 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:40.967797 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8t6s\" (UniqueName: \"kubernetes.io/projected/b072b11d-bfb0-432e-b5d6-65d887f58cc4-kube-api-access-q8t6s\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:40.967894 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:40.967830 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcbcf982-571a-4281-ac51-d6c5fb9592bc-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:40.967894 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:40.967843 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m9lx\" (UniqueName: \"kubernetes.io/projected/fcbcf982-571a-4281-ac51-d6c5fb9592bc-kube-api-access-5m9lx\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:40.967894 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:40.967855 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b072b11d-bfb0-432e-b5d6-65d887f58cc4-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:41.053617 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:41.053580 4947 generic.go:356] "Generic (PLEG): container finished" podID="fcbcf982-571a-4281-ac51-d6c5fb9592bc" containerID="62426d31bdc6aec14760cb1a9119610833e864250ca285ee85e249390fa2f463" exitCode=0 Nov 03 16:44:41.053853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:41.053631 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx" Nov 03 16:44:41.053853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:41.053668 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx" event={"ID":"fcbcf982-571a-4281-ac51-d6c5fb9592bc","Type":"ContainerDied","Data":"62426d31bdc6aec14760cb1a9119610833e864250ca285ee85e249390fa2f463"} Nov 03 16:44:41.053853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:41.053746 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx" event={"ID":"fcbcf982-571a-4281-ac51-d6c5fb9592bc","Type":"ContainerDied","Data":"baac0d845c6a5e0cc6db605f43b6adb0234baaedbc8efafe771f6a3a414cc94d"} Nov 03 16:44:41.053853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:41.053777 4947 scope.go:122] "RemoveContainer" containerID="62426d31bdc6aec14760cb1a9119610833e864250ca285ee85e249390fa2f463" Nov 03 16:44:41.055384 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:41.055358 4947 generic.go:356] "Generic (PLEG): container finished" podID="b072b11d-bfb0-432e-b5d6-65d887f58cc4" containerID="2644b54ba59bde1b31e58ff306b57ffb936917933d9138457ddaac2d671bfdfa" exitCode=0 Nov 03 16:44:41.055513 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:41.055411 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt" Nov 03 16:44:41.055513 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:41.055405 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt" event={"ID":"b072b11d-bfb0-432e-b5d6-65d887f58cc4","Type":"ContainerDied","Data":"2644b54ba59bde1b31e58ff306b57ffb936917933d9138457ddaac2d671bfdfa"} Nov 03 16:44:41.055513 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:41.055468 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt" event={"ID":"b072b11d-bfb0-432e-b5d6-65d887f58cc4","Type":"ContainerDied","Data":"19c6753dcf97f3d842f37c553f9c84c3248ff1083c8fd30979d91a27b6b0c230"} Nov 03 16:44:41.069097 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:41.069072 4947 scope.go:122] "RemoveContainer" containerID="62426d31bdc6aec14760cb1a9119610833e864250ca285ee85e249390fa2f463" Nov 03 16:44:41.069455 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:44:41.069423 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62426d31bdc6aec14760cb1a9119610833e864250ca285ee85e249390fa2f463\": container with ID starting with 62426d31bdc6aec14760cb1a9119610833e864250ca285ee85e249390fa2f463 not found: ID does not exist" containerID="62426d31bdc6aec14760cb1a9119610833e864250ca285ee85e249390fa2f463" Nov 03 16:44:41.069564 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:41.069476 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62426d31bdc6aec14760cb1a9119610833e864250ca285ee85e249390fa2f463"} err="failed to get container status \"62426d31bdc6aec14760cb1a9119610833e864250ca285ee85e249390fa2f463\": rpc error: code = NotFound desc = could not find container \"62426d31bdc6aec14760cb1a9119610833e864250ca285ee85e249390fa2f463\": container with ID starting with 62426d31bdc6aec14760cb1a9119610833e864250ca285ee85e249390fa2f463 not found: ID does not exist" Nov 03 16:44:41.069564 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:41.069500 4947 scope.go:122] "RemoveContainer" containerID="2644b54ba59bde1b31e58ff306b57ffb936917933d9138457ddaac2d671bfdfa" Nov 03 16:44:41.080281 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:41.080253 4947 scope.go:122] "RemoveContainer" containerID="2644b54ba59bde1b31e58ff306b57ffb936917933d9138457ddaac2d671bfdfa" Nov 03 16:44:41.080611 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:44:41.080590 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2644b54ba59bde1b31e58ff306b57ffb936917933d9138457ddaac2d671bfdfa\": container with ID starting with 2644b54ba59bde1b31e58ff306b57ffb936917933d9138457ddaac2d671bfdfa not found: ID does not exist" containerID="2644b54ba59bde1b31e58ff306b57ffb936917933d9138457ddaac2d671bfdfa" Nov 03 16:44:41.080675 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:41.080618 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2644b54ba59bde1b31e58ff306b57ffb936917933d9138457ddaac2d671bfdfa"} err="failed to get container status \"2644b54ba59bde1b31e58ff306b57ffb936917933d9138457ddaac2d671bfdfa\": rpc error: code = NotFound desc = could not find container \"2644b54ba59bde1b31e58ff306b57ffb936917933d9138457ddaac2d671bfdfa\": container with ID starting with 2644b54ba59bde1b31e58ff306b57ffb936917933d9138457ddaac2d671bfdfa not found: ID does not exist" Nov 03 16:44:41.112478 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:41.112439 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx"] Nov 03 16:44:41.122802 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:41.120175 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-744fb7474c-kgxbx"] Nov 03 16:44:41.159225 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:41.159185 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt"] Nov 03 16:44:41.164497 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:41.164462 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-156c2a-874648cc-hfdlt"] Nov 03 16:44:42.568922 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:42.568887 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b072b11d-bfb0-432e-b5d6-65d887f58cc4" path="/var/lib/kubelet/pods/b072b11d-bfb0-432e-b5d6-65d887f58cc4/volumes" Nov 03 16:44:42.569597 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:42.569570 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcbcf982-571a-4281-ac51-d6c5fb9592bc" path="/var/lib/kubelet/pods/fcbcf982-571a-4281-ac51-d6c5fb9592bc/volumes" Nov 03 16:44:42.627428 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:42.627383 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv"] Nov 03 16:44:42.627672 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:42.627641 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" podUID="0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" containerName="istio-proxy" containerID="cri-o://f218af63a4c82a7ed8a633bf9979802cb2ad68eb90403ddcabe210bde9adda55" gracePeriod=30 Nov 03 16:44:44.170551 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.170512 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm"] Nov 03 16:44:44.171592 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.171568 4947 cpu_manager.go:414] "containerMap: removing container" podUID="fcbcf982-571a-4281-ac51-d6c5fb9592bc" containerName="payload-pre-processing" Nov 03 16:44:44.171689 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.171596 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcbcf982-571a-4281-ac51-d6c5fb9592bc" containerName="payload-pre-processing" Nov 03 16:44:44.171689 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.171678 4947 cpu_manager.go:414] "containerMap: removing container" podUID="b072b11d-bfb0-432e-b5d6-65d887f58cc4" containerName="payload-processing" Nov 03 16:44:44.171689 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.171687 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b072b11d-bfb0-432e-b5d6-65d887f58cc4" containerName="payload-processing" Nov 03 16:44:44.171895 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.171879 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="b072b11d-bfb0-432e-b5d6-65d887f58cc4" containerName="payload-processing" Nov 03 16:44:44.171938 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.171903 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcbcf982-571a-4281-ac51-d6c5fb9592bc" containerName="payload-pre-processing" Nov 03 16:44:44.177923 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.177890 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.209841 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.209809 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2cb1bb-openshift-default-dockercfg-qbn7q\"" Nov 03 16:44:44.214841 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.214808 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm"] Nov 03 16:44:44.236123 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.236079 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-data\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.236123 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.236122 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-podinfo\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.236407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.236139 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-envoy\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.236407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.236193 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3757c5e0-0c52-4c99-a008-d6d944b33983-istiod-ca-cert\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.236407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.236227 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-token\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.236407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.236251 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-workload-socket\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.236407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.236296 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-credential-socket\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.236407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.236312 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxgkv\" (UniqueName: \"kubernetes.io/projected/3757c5e0-0c52-4c99-a008-d6d944b33983-kube-api-access-zxgkv\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.236407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.236340 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-workload-certs\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.338564 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.338522 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3757c5e0-0c52-4c99-a008-d6d944b33983-istiod-ca-cert\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.338906 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.338879 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-token\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.339046 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.339031 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-workload-socket\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.339212 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.339197 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-credential-socket\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.339319 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.339302 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxgkv\" (UniqueName: \"kubernetes.io/projected/3757c5e0-0c52-4c99-a008-d6d944b33983-kube-api-access-zxgkv\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.339450 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.339434 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-workload-certs\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.339777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.339756 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-data\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.339923 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.339906 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-podinfo\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.340035 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.340021 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-envoy\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.340150 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.340129 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-workload-socket\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.368813 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.340479 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-workload-certs\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.369061 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.340724 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-credential-socket\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.369162 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.341241 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-data\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.369249 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.350447 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3757c5e0-0c52-4c99-a008-d6d944b33983-istiod-ca-cert\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.401736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.396234 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-token\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.410569 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.410495 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxgkv\" (UniqueName: \"kubernetes.io/projected/3757c5e0-0c52-4c99-a008-d6d944b33983-kube-api-access-zxgkv\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.427791 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.425633 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-envoy\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.428578 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.428552 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-podinfo\") pod \"e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.502735 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.502091 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:44.960753 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.959754 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm"] Nov 03 16:44:44.975964 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.974037 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:44:44.975964 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.974143 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:44:44.975964 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:44.974262 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:44:45.089927 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:45.089889 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" event={"ID":"3757c5e0-0c52-4c99-a008-d6d944b33983","Type":"ContainerStarted","Data":"295adc1234281ab4b599907511e063eb871c1e040e4620f7b5d77a1f62800738"} Nov 03 16:44:45.090108 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:45.089955 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" event={"ID":"3757c5e0-0c52-4c99-a008-d6d944b33983","Type":"ContainerStarted","Data":"3e0e2ab6d0f53745ed60e935a132fd50d2469f797cd419c38a573f72069be5e6"} Nov 03 16:44:45.152569 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:45.152499 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" podStartSLOduration=1.152478976 podStartE2EDuration="1.152478976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:45.149484462 +0000 UTC m=+1828.738484434" watchObservedRunningTime="2026-11-03 16:44:45.152478976 +0000 UTC m=+1828.741478946" Nov 03 16:44:45.503563 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:45.503522 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:46.019795 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.019724 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" podUID="416cce8d-b3ae-4b52-adbb-e33fe9ec04f8" containerName="main" containerID="cri-o://8f9ba78da88a969087d405bbb6467967a83f50193a8652b3f920110c3ac024ef" gracePeriod=45 Nov 03 16:44:46.305392 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.305359 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:44:46.308634 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.308610 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-kube-api-access-gstk2\" (UniqueName: \"kubernetes.io/projected/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-kube-api-access-gstk2\") pod \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " Nov 03 16:44:46.308763 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.308750 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-model-cache\") pod \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " Nov 03 16:44:46.308831 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.308799 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-dshm\") pod \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " Nov 03 16:44:46.308887 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.308857 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-tls-certs\") pod \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " Nov 03 16:44:46.309049 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.309016 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-model-cache" pod "416cce8d-b3ae-4b52-adbb-e33fe9ec04f8" (UID: "416cce8d-b3ae-4b52-adbb-e33fe9ec04f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:46.309049 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.309028 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-home\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-home\") pod \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " Nov 03 16:44:46.309248 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.309109 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-tmp-dir\") pod \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\" (UID: \"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8\") " Nov 03 16:44:46.309248 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.309227 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-home" pod "416cce8d-b3ae-4b52-adbb-e33fe9ec04f8" (UID: "416cce8d-b3ae-4b52-adbb-e33fe9ec04f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:46.309570 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.309529 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-tmp-dir" pod "416cce8d-b3ae-4b52-adbb-e33fe9ec04f8" (UID: "416cce8d-b3ae-4b52-adbb-e33fe9ec04f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:46.310189 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.310165 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:46.310306 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.310194 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:46.310306 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.310208 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:46.311222 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.311198 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-kube-api-access-gstk2" pod "416cce8d-b3ae-4b52-adbb-e33fe9ec04f8" (UID: "416cce8d-b3ae-4b52-adbb-e33fe9ec04f8"). InnerVolumeSpecName "kube-api-access-gstk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:46.311380 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.311358 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-tls-certs" pod "416cce8d-b3ae-4b52-adbb-e33fe9ec04f8" (UID: "416cce8d-b3ae-4b52-adbb-e33fe9ec04f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:44:46.311658 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.311633 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-dshm" pod "416cce8d-b3ae-4b52-adbb-e33fe9ec04f8" (UID: "416cce8d-b3ae-4b52-adbb-e33fe9ec04f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:46.412199 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.412166 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gstk2\" (UniqueName: \"kubernetes.io/projected/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-kube-api-access-gstk2\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:46.412199 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.412197 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:46.412416 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.412209 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:46.509926 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:46.509900 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:47.104993 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:47.104948 4947 generic.go:356] "Generic (PLEG): container finished" podID="416cce8d-b3ae-4b52-adbb-e33fe9ec04f8" containerID="8f9ba78da88a969087d405bbb6467967a83f50193a8652b3f920110c3ac024ef" exitCode=0 Nov 03 16:44:47.105202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:47.105002 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" Nov 03 16:44:47.105202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:47.105047 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" event={"ID":"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8","Type":"ContainerDied","Data":"8f9ba78da88a969087d405bbb6467967a83f50193a8652b3f920110c3ac024ef"} Nov 03 16:44:47.105202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:47.105093 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2" event={"ID":"416cce8d-b3ae-4b52-adbb-e33fe9ec04f8","Type":"ContainerDied","Data":"334dcca0a1a1008a2d89b7407ab1467de93f88a4b8c47b663fbe323616e41c77"} Nov 03 16:44:47.105202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:47.105119 4947 scope.go:122] "RemoveContainer" containerID="8f9ba78da88a969087d405bbb6467967a83f50193a8652b3f920110c3ac024ef" Nov 03 16:44:47.105595 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:47.105573 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:47.109078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:47.109056 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:44:47.116313 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:47.116292 4947 scope.go:122] "RemoveContainer" containerID="8f9ba78da88a969087d405bbb6467967a83f50193a8652b3f920110c3ac024ef" Nov 03 16:44:47.116676 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:44:47.116645 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9ba78da88a969087d405bbb6467967a83f50193a8652b3f920110c3ac024ef\": container with ID starting with 8f9ba78da88a969087d405bbb6467967a83f50193a8652b3f920110c3ac024ef not found: ID does not exist" containerID="8f9ba78da88a969087d405bbb6467967a83f50193a8652b3f920110c3ac024ef" Nov 03 16:44:47.116782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:47.116719 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9ba78da88a969087d405bbb6467967a83f50193a8652b3f920110c3ac024ef"} err="failed to get container status \"8f9ba78da88a969087d405bbb6467967a83f50193a8652b3f920110c3ac024ef\": rpc error: code = NotFound desc = could not find container \"8f9ba78da88a969087d405bbb6467967a83f50193a8652b3f920110c3ac024ef\": container with ID starting with 8f9ba78da88a969087d405bbb6467967a83f50193a8652b3f920110c3ac024ef not found: ID does not exist" Nov 03 16:44:47.261770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:47.261735 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2"] Nov 03 16:44:47.268395 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:47.268366 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-8650a6e8/e2e-lifecycle-model-8650a6e8-kserve-5f467c4dd9-gwmx2"] Nov 03 16:44:47.947618 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:47.947593 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:44:48.056789 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.051653 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-workload-socket\") pod \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " Nov 03 16:44:48.056789 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.051732 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-envoy\") pod \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " Nov 03 16:44:48.056789 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.051864 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-data\") pod \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " Nov 03 16:44:48.056789 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.051894 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-workload-certs\") pod \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " Nov 03 16:44:48.056789 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.051987 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-token\" (UniqueName: \"kubernetes.io/projected/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-token\") pod \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " Nov 03 16:44:48.056789 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.052047 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istiod-ca-cert\") pod \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " Nov 03 16:44:48.056789 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.052079 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-podinfo\") pod \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " Nov 03 16:44:48.056789 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.052131 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-kube-api-access-7zhd9\" (UniqueName: \"kubernetes.io/projected/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-kube-api-access-7zhd9\") pod \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " Nov 03 16:44:48.056789 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.052282 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-credential-socket\") pod \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\" (UID: \"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3\") " Nov 03 16:44:48.058855 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.058798 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-credential-socket" pod "0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" (UID: "0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:48.067222 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.067176 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-workload-socket" pod "0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" (UID: "0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:48.067547 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.067521 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istiod-ca-cert" pod "0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" (UID: "0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:44:48.067810 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.067785 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-data" pod "0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" (UID: "0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:48.093722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.068334 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-workload-certs" pod "0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" (UID: "0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:48.101731 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.100094 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-token" pod "0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" (UID: "0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:48.101731 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.100778 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-kube-api-access-7zhd9" pod "0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" (UID: "0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3"). InnerVolumeSpecName "kube-api-access-7zhd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:44:48.119790 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.113234 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-podinfo" pod "0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" (UID: "0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:44:48.140733 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.138860 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-envoy" pod "0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" (UID: "0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:44:48.199382 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.198419 4947 generic.go:356] "Generic (PLEG): container finished" podID="0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" containerID="f218af63a4c82a7ed8a633bf9979802cb2ad68eb90403ddcabe210bde9adda55" exitCode=0 Nov 03 16:44:48.199382 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.198718 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" event={"ID":"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3","Type":"ContainerDied","Data":"f218af63a4c82a7ed8a633bf9979802cb2ad68eb90403ddcabe210bde9adda55"} Nov 03 16:44:48.199382 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.198781 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" event={"ID":"0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3","Type":"ContainerDied","Data":"b3561a375be564ed8a7dadb0b88c41f3ed195c94f0184ec37d76a8d0afcc96cb"} Nov 03 16:44:48.199382 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.198813 4947 scope.go:122] "RemoveContainer" containerID="f218af63a4c82a7ed8a633bf9979802cb2ad68eb90403ddcabe210bde9adda55" Nov 03 16:44:48.199382 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.198945 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv" Nov 03 16:44:48.242377 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.242343 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:48.328728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.324640 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:48.328728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.325949 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:48.328728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.325969 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:48.328728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.325982 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:48.328728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.325995 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:48.328728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.326023 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:48.328728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.326037 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:48.328728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.326052 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zhd9\" (UniqueName: \"kubernetes.io/projected/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3-kube-api-access-7zhd9\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:44:48.349721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.346976 4947 scope.go:122] "RemoveContainer" containerID="5dba12412ca78a005f1d5e2cb6829c1d43d13f2f0af966da3d7860b6b2e17266" Nov 03 16:44:48.372779 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.372749 4947 scope.go:122] "RemoveContainer" containerID="f218af63a4c82a7ed8a633bf9979802cb2ad68eb90403ddcabe210bde9adda55" Nov 03 16:44:48.375884 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:44:48.375850 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f218af63a4c82a7ed8a633bf9979802cb2ad68eb90403ddcabe210bde9adda55\": container with ID starting with f218af63a4c82a7ed8a633bf9979802cb2ad68eb90403ddcabe210bde9adda55 not found: ID does not exist" containerID="f218af63a4c82a7ed8a633bf9979802cb2ad68eb90403ddcabe210bde9adda55" Nov 03 16:44:48.376016 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.375895 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f218af63a4c82a7ed8a633bf9979802cb2ad68eb90403ddcabe210bde9adda55"} err="failed to get container status \"f218af63a4c82a7ed8a633bf9979802cb2ad68eb90403ddcabe210bde9adda55\": rpc error: code = NotFound desc = could not find container \"f218af63a4c82a7ed8a633bf9979802cb2ad68eb90403ddcabe210bde9adda55\": container with ID starting with f218af63a4c82a7ed8a633bf9979802cb2ad68eb90403ddcabe210bde9adda55 not found: ID does not exist" Nov 03 16:44:48.376016 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.375915 4947 scope.go:122] "RemoveContainer" containerID="5dba12412ca78a005f1d5e2cb6829c1d43d13f2f0af966da3d7860b6b2e17266" Nov 03 16:44:48.376735 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:44:48.376667 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dba12412ca78a005f1d5e2cb6829c1d43d13f2f0af966da3d7860b6b2e17266\": container with ID starting with 5dba12412ca78a005f1d5e2cb6829c1d43d13f2f0af966da3d7860b6b2e17266 not found: ID does not exist" containerID="5dba12412ca78a005f1d5e2cb6829c1d43d13f2f0af966da3d7860b6b2e17266" Nov 03 16:44:48.376842 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.376745 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dba12412ca78a005f1d5e2cb6829c1d43d13f2f0af966da3d7860b6b2e17266"} err="failed to get container status \"5dba12412ca78a005f1d5e2cb6829c1d43d13f2f0af966da3d7860b6b2e17266\": rpc error: code = NotFound desc = could not find container \"5dba12412ca78a005f1d5e2cb6829c1d43d13f2f0af966da3d7860b6b2e17266\": container with ID starting with 5dba12412ca78a005f1d5e2cb6829c1d43d13f2f0af966da3d7860b6b2e17266 not found: ID does not exist" Nov 03 16:44:48.408236 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.408199 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv"] Nov 03 16:44:48.413593 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.413569 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4e3be7-openshift-default-858846ff75-lmfxv"] Nov 03 16:44:48.569121 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.569075 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" path="/var/lib/kubelet/pods/0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3/volumes" Nov 03 16:44:48.569633 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:48.569616 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="416cce8d-b3ae-4b52-adbb-e33fe9ec04f8" path="/var/lib/kubelet/pods/416cce8d-b3ae-4b52-adbb-e33fe9ec04f8/volumes" Nov 03 16:44:49.941024 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:49.940985 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw"] Nov 03 16:44:49.941754 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:49.941736 4947 cpu_manager.go:414] "containerMap: removing container" podUID="0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" containerName="istio-proxy" Nov 03 16:44:49.941754 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:49.941753 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" containerName="istio-proxy" Nov 03 16:44:49.941903 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:49.941768 4947 cpu_manager.go:414] "containerMap: removing container" podUID="416cce8d-b3ae-4b52-adbb-e33fe9ec04f8" containerName="main" Nov 03 16:44:49.941903 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:49.941773 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416cce8d-b3ae-4b52-adbb-e33fe9ec04f8" containerName="main" Nov 03 16:44:49.941903 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:49.941796 4947 cpu_manager.go:414] "containerMap: removing container" podUID="0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" containerName="istio-proxy" Nov 03 16:44:49.941903 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:49.941801 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" containerName="istio-proxy" Nov 03 16:44:49.942024 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:49.941912 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" containerName="istio-proxy" Nov 03 16:44:49.942024 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:49.941927 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="416cce8d-b3ae-4b52-adbb-e33fe9ec04f8" containerName="main" Nov 03 16:44:49.947264 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:49.947243 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw" Nov 03 16:44:49.977409 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:49.977372 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw"] Nov 03 16:44:49.986403 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:49.986366 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2cb1bb\"" Nov 03 16:44:49.991901 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:49.991866 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489"] Nov 03 16:44:49.992732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:49.992708 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c0cbf00-cde2-4e77-a8f1-5bf936b6a0f3" containerName="istio-proxy" Nov 03 16:44:49.996892 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:49.996787 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489" Nov 03 16:44:50.016905 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:50.016868 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2cb1bb-dockercfg-wmh22\"" Nov 03 16:44:50.036046 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:50.036007 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489"] Nov 03 16:44:50.052978 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:50.052942 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cfbb6cd-2794-4071-b870-fd4bd0dc97ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489\" (UID: \"6cfbb6cd-2794-4071-b870-fd4bd0dc97ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489" Nov 03 16:44:50.053162 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:50.053053 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbggk\" (UniqueName: \"kubernetes.io/projected/8e1e60eb-8bf7-4d39-998c-d5e488170f41-kube-api-access-xbggk\") pod \"payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw\" (UID: \"8e1e60eb-8bf7-4d39-998c-d5e488170f41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw" Nov 03 16:44:50.053239 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:50.053203 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh98t\" (UniqueName: \"kubernetes.io/projected/6cfbb6cd-2794-4071-b870-fd4bd0dc97ab-kube-api-access-wh98t\") pod \"payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489\" (UID: \"6cfbb6cd-2794-4071-b870-fd4bd0dc97ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489" Nov 03 16:44:50.053239 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:50.053231 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1e60eb-8bf7-4d39-998c-d5e488170f41-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw\" (UID: \"8e1e60eb-8bf7-4d39-998c-d5e488170f41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw" Nov 03 16:44:50.202776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:50.189538 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cfbb6cd-2794-4071-b870-fd4bd0dc97ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489\" (UID: \"6cfbb6cd-2794-4071-b870-fd4bd0dc97ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489" Nov 03 16:44:50.202776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:50.189662 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbggk\" (UniqueName: \"kubernetes.io/projected/8e1e60eb-8bf7-4d39-998c-d5e488170f41-kube-api-access-xbggk\") pod \"payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw\" (UID: \"8e1e60eb-8bf7-4d39-998c-d5e488170f41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw" Nov 03 16:44:50.202776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:50.189854 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh98t\" (UniqueName: \"kubernetes.io/projected/6cfbb6cd-2794-4071-b870-fd4bd0dc97ab-kube-api-access-wh98t\") pod \"payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489\" (UID: \"6cfbb6cd-2794-4071-b870-fd4bd0dc97ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489" Nov 03 16:44:50.202776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:50.189891 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1e60eb-8bf7-4d39-998c-d5e488170f41-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw\" (UID: \"8e1e60eb-8bf7-4d39-998c-d5e488170f41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw" Nov 03 16:44:50.202776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:50.191061 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1e60eb-8bf7-4d39-998c-d5e488170f41-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw\" (UID: \"8e1e60eb-8bf7-4d39-998c-d5e488170f41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw" Nov 03 16:44:50.202776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:50.191830 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cfbb6cd-2794-4071-b870-fd4bd0dc97ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489\" (UID: \"6cfbb6cd-2794-4071-b870-fd4bd0dc97ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489" Nov 03 16:44:50.243510 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:50.243472 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbggk\" (UniqueName: \"kubernetes.io/projected/8e1e60eb-8bf7-4d39-998c-d5e488170f41-kube-api-access-xbggk\") pod \"payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw\" (UID: \"8e1e60eb-8bf7-4d39-998c-d5e488170f41\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw" Nov 03 16:44:50.262858 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:50.262824 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw" Nov 03 16:44:50.264207 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:50.263200 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh98t\" (UniqueName: \"kubernetes.io/projected/6cfbb6cd-2794-4071-b870-fd4bd0dc97ab-kube-api-access-wh98t\") pod \"payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489\" (UID: \"6cfbb6cd-2794-4071-b870-fd4bd0dc97ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489" Nov 03 16:44:50.318158 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:50.318115 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489" Nov 03 16:44:50.900724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:50.899442 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw"] Nov 03 16:44:50.989840 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:50.982282 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489"] Nov 03 16:44:51.385025 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:51.384933 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw" event={"ID":"8e1e60eb-8bf7-4d39-998c-d5e488170f41","Type":"ContainerStarted","Data":"cef5beb2fdeb9603a12f3d01545e950b7e5d6dec667d6d31a4cde65cc061aab3"} Nov 03 16:44:51.392910 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:51.392685 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw" event={"ID":"8e1e60eb-8bf7-4d39-998c-d5e488170f41","Type":"ContainerStarted","Data":"207d70ef3427a266dcd6447b2d9aa4dc4aa1da6d5401b227354f6fa79a24b5f2"} Nov 03 16:44:51.396727 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:51.393162 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw" Nov 03 16:44:51.402044 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:51.402013 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489" event={"ID":"6cfbb6cd-2794-4071-b870-fd4bd0dc97ab","Type":"ContainerStarted","Data":"46dcf2588ce614a70abbe211628e5133f22269f8035d6dfabbba41d8da3c2844"} Nov 03 16:44:51.403254 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:51.403233 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489" Nov 03 16:44:51.403449 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:51.403430 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489" event={"ID":"6cfbb6cd-2794-4071-b870-fd4bd0dc97ab","Type":"ContainerStarted","Data":"a17b3919068edbfb45a8d523dba4d74adf34c4c97520c220093c5b3f9ccc48c0"} Nov 03 16:44:51.454944 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:51.454883 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw" podStartSLOduration=2.454863987 podStartE2EDuration="2.454863987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:51.453711701 +0000 UTC m=+1835.042711660" watchObservedRunningTime="2026-11-03 16:44:51.454863987 +0000 UTC m=+1835.043863958" Nov 03 16:44:51.542416 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:51.542363 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489" podStartSLOduration=2.542348535 podStartE2EDuration="2.542348535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:51.540225976 +0000 UTC m=+1835.129225944" watchObservedRunningTime="2026-11-03 16:44:51.542348535 +0000 UTC m=+1835.131348505" Nov 03 16:44:53.659453 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.659419 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst"] Nov 03 16:44:53.664372 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.664342 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.681390 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.681353 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2aa2ad-openshift-default-dockercfg-jcgnq\"" Nov 03 16:44:53.699677 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.699643 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst"] Nov 03 16:44:53.724171 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.724135 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istiod-ca-cert\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.724399 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.724184 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-workload-certs\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.724399 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.724262 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-data\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.724531 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.724410 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-envoy\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.724531 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.724509 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-podinfo\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.724640 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.724543 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-credential-socket\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.724721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.724648 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-token\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.724721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.724675 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct8ln\" (UniqueName: \"kubernetes.io/projected/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-kube-api-access-ct8ln\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.724844 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.724746 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-workload-socket\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.849724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.826846 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-data\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.850060 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.833223 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-data\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.850498 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.850438 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-envoy\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.877052 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.877017 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-podinfo\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.877384 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.877357 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-credential-socket\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.877876 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.877813 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-token\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.878170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.878143 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct8ln\" (UniqueName: \"kubernetes.io/projected/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-kube-api-access-ct8ln\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.878277 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.878250 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-workload-socket\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.878344 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.878295 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istiod-ca-cert\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.878405 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.878392 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-workload-certs\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.879401 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.878975 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-workload-certs\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.879401 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.879356 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-workload-socket\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.880337 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.880076 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istiod-ca-cert\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.880337 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.880304 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-credential-socket\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.883043 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.883016 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-envoy\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.917780 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.913244 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-podinfo\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.930956 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.928676 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-token\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.930956 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.930110 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct8ln\" (UniqueName: \"kubernetes.io/projected/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-kube-api-access-ct8ln\") pod \"e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:53.991146 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:53.991054 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:54.517338 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:44:54.517273 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5849aa_2bda_4812_bdcc_39bfb3be4ee3.slice/crio-52f68beeb39211984060a69a3b72a26a9323a6c10052d5cca701a1b85d8a7961 WatchSource:0}: Error finding container 52f68beeb39211984060a69a3b72a26a9323a6c10052d5cca701a1b85d8a7961: Status 404 returned error can't find the container with id 52f68beeb39211984060a69a3b72a26a9323a6c10052d5cca701a1b85d8a7961 Nov 03 16:44:54.532239 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:54.532180 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:44:54.555110 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:54.555063 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:44:54.556469 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:54.556428 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:44:54.644879 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:54.644839 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst"] Nov 03 16:44:55.228962 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:55.225771 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd"] Nov 03 16:44:55.236388 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:55.234239 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd" Nov 03 16:44:55.239826 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:55.239690 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd"] Nov 03 16:44:55.272092 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:55.272056 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-156c2a\"" Nov 03 16:44:55.272652 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:55.272419 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-156c2a-dockercfg-bnpps\"" Nov 03 16:44:55.310239 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:55.309947 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85g24\" (UniqueName: \"kubernetes.io/projected/541df157-b8b7-4031-a272-c5a80899ae76-kube-api-access-85g24\") pod \"payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd\" (UID: \"541df157-b8b7-4031-a272-c5a80899ae76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd" Nov 03 16:44:55.310239 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:55.310021 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/541df157-b8b7-4031-a272-c5a80899ae76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd\" (UID: \"541df157-b8b7-4031-a272-c5a80899ae76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd" Nov 03 16:44:55.411659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:55.411624 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85g24\" (UniqueName: \"kubernetes.io/projected/541df157-b8b7-4031-a272-c5a80899ae76-kube-api-access-85g24\") pod \"payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd\" (UID: \"541df157-b8b7-4031-a272-c5a80899ae76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd" Nov 03 16:44:55.411894 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:55.411689 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/541df157-b8b7-4031-a272-c5a80899ae76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd\" (UID: \"541df157-b8b7-4031-a272-c5a80899ae76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd" Nov 03 16:44:55.412522 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:55.412502 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/541df157-b8b7-4031-a272-c5a80899ae76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd\" (UID: \"541df157-b8b7-4031-a272-c5a80899ae76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd" Nov 03 16:44:55.437194 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:55.437156 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85g24\" (UniqueName: \"kubernetes.io/projected/541df157-b8b7-4031-a272-c5a80899ae76-kube-api-access-85g24\") pod \"payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd\" (UID: \"541df157-b8b7-4031-a272-c5a80899ae76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd" Nov 03 16:44:55.444895 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:55.444857 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" event={"ID":"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3","Type":"ContainerStarted","Data":"844fe8474c8538f55ea779a4f4dac92103f6697d3fd7d374ef2034c566cc27a5"} Nov 03 16:44:55.445031 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:55.444912 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" event={"ID":"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3","Type":"ContainerStarted","Data":"52f68beeb39211984060a69a3b72a26a9323a6c10052d5cca701a1b85d8a7961"} Nov 03 16:44:55.495870 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:55.495811 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" podStartSLOduration=2.495793119 podStartE2EDuration="2.495793119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:55.493278423 +0000 UTC m=+1839.082278392" watchObservedRunningTime="2026-11-03 16:44:55.495793119 +0000 UTC m=+1839.084793087" Nov 03 16:44:55.619760 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:55.610970 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd" Nov 03 16:44:55.994720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:55.991533 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:56.030808 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:56.029139 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:56.065926 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:56.065885 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd"] Nov 03 16:44:56.067123 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:44:56.067093 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541df157_b8b7_4031_a272_c5a80899ae76.slice/crio-4ae37b994295ae78696f74b72af204127faf4ad280728688f034ea9a624e2ef3 WatchSource:0}: Error finding container 4ae37b994295ae78696f74b72af204127faf4ad280728688f034ea9a624e2ef3: Status 404 returned error can't find the container with id 4ae37b994295ae78696f74b72af204127faf4ad280728688f034ea9a624e2ef3 Nov 03 16:44:56.456809 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:56.456685 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd" event={"ID":"541df157-b8b7-4031-a272-c5a80899ae76","Type":"ContainerStarted","Data":"4ae37b994295ae78696f74b72af204127faf4ad280728688f034ea9a624e2ef3"} Nov 03 16:44:56.457647 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:56.457231 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:56.458525 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:56.458499 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:44:58.504387 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:58.504349 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd" event={"ID":"541df157-b8b7-4031-a272-c5a80899ae76","Type":"ContainerStarted","Data":"90eec019b84bf8fea5e281ee4f458810bcfd3f38fccfd02c52b98be94a1bdb49"} Nov 03 16:44:58.504838 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:58.504574 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd" Nov 03 16:44:58.564652 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:44:58.564599 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd" podStartSLOduration=2.055796594 podStartE2EDuration="3.564583974s" totalImagesPullingTime="1.50878738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:44:58.563230507 +0000 UTC m=+1842.152230479" watchObservedRunningTime="2026-11-03 16:44:58.564583974 +0000 UTC m=+1842.153583943" Nov 03 16:45:00.132007 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:45:00.131963 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:45:03.428144 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:03.428109 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw" Nov 03 16:45:03.428628 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:03.428213 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489" Nov 03 16:45:05.695996 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:05.695964 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m"] Nov 03 16:45:05.750032 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:05.750000 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8"] Nov 03 16:45:07.338053 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:07.338008 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:45:08.729584 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:08.729549 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h"] Nov 03 16:45:08.774451 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:08.774421 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x"] Nov 03 16:45:09.526281 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:09.526251 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd" Nov 03 16:45:10.697129 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:10.697075 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m" podUID="96672632-057a-4ca2-ae04-f5d429f470a7" containerName="payload-processing" containerID="cri-o://4f9b185b06d1c5750e06f61788ed9836feac27a33048134b8f50b2cda956cb38" gracePeriod=25 Nov 03 16:45:10.751202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:10.751167 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8" podUID="1548c8a6-c110-4168-9568-5adcc07df255" containerName="payload-pre-processing" containerID="cri-o://fe3693c109056f38f582e883703a09beeb5548a0b753704b738c84f703ff19ac" gracePeriod=25 Nov 03 16:45:11.052969 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.052938 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m" Nov 03 16:45:11.110620 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.110591 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8" Nov 03 16:45:11.201137 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.201099 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96672632-057a-4ca2-ae04-f5d429f470a7-kube-api-access-9f29q\" (UniqueName: \"kubernetes.io/projected/96672632-057a-4ca2-ae04-f5d429f470a7-kube-api-access-9f29q\") pod \"96672632-057a-4ca2-ae04-f5d429f470a7\" (UID: \"96672632-057a-4ca2-ae04-f5d429f470a7\") " Nov 03 16:45:11.201341 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.201294 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1548c8a6-c110-4168-9568-5adcc07df255-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1548c8a6-c110-4168-9568-5adcc07df255-plugins-config-volume\") pod \"1548c8a6-c110-4168-9568-5adcc07df255\" (UID: \"1548c8a6-c110-4168-9568-5adcc07df255\") " Nov 03 16:45:11.201410 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.201349 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96672632-057a-4ca2-ae04-f5d429f470a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96672632-057a-4ca2-ae04-f5d429f470a7-plugins-config-volume\") pod \"96672632-057a-4ca2-ae04-f5d429f470a7\" (UID: \"96672632-057a-4ca2-ae04-f5d429f470a7\") " Nov 03 16:45:11.201460 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.201408 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1548c8a6-c110-4168-9568-5adcc07df255-kube-api-access-mgjth\" (UniqueName: \"kubernetes.io/projected/1548c8a6-c110-4168-9568-5adcc07df255-kube-api-access-mgjth\") pod \"1548c8a6-c110-4168-9568-5adcc07df255\" (UID: \"1548c8a6-c110-4168-9568-5adcc07df255\") " Nov 03 16:45:11.201744 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.201687 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1548c8a6-c110-4168-9568-5adcc07df255-plugins-config-volume" pod "1548c8a6-c110-4168-9568-5adcc07df255" (UID: "1548c8a6-c110-4168-9568-5adcc07df255"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:11.201906 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.201822 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96672632-057a-4ca2-ae04-f5d429f470a7-plugins-config-volume" pod "96672632-057a-4ca2-ae04-f5d429f470a7" (UID: "96672632-057a-4ca2-ae04-f5d429f470a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:11.202164 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.202144 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1548c8a6-c110-4168-9568-5adcc07df255-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:11.203652 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.203627 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96672632-057a-4ca2-ae04-f5d429f470a7-kube-api-access-9f29q" pod "96672632-057a-4ca2-ae04-f5d429f470a7" (UID: "96672632-057a-4ca2-ae04-f5d429f470a7"). InnerVolumeSpecName "kube-api-access-9f29q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:11.204304 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.204279 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1548c8a6-c110-4168-9568-5adcc07df255-kube-api-access-mgjth" pod "1548c8a6-c110-4168-9568-5adcc07df255" (UID: "1548c8a6-c110-4168-9568-5adcc07df255"). InnerVolumeSpecName "kube-api-access-mgjth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:11.303626 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.303588 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96672632-057a-4ca2-ae04-f5d429f470a7-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:11.303626 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.303621 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgjth\" (UniqueName: \"kubernetes.io/projected/1548c8a6-c110-4168-9568-5adcc07df255-kube-api-access-mgjth\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:11.303626 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.303631 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f29q\" (UniqueName: \"kubernetes.io/projected/96672632-057a-4ca2-ae04-f5d429f470a7-kube-api-access-9f29q\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:11.802387 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.802350 4947 generic.go:356] "Generic (PLEG): container finished" podID="96672632-057a-4ca2-ae04-f5d429f470a7" containerID="4f9b185b06d1c5750e06f61788ed9836feac27a33048134b8f50b2cda956cb38" exitCode=0 Nov 03 16:45:11.802932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.802393 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m" Nov 03 16:45:11.802932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.802438 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m" event={"ID":"96672632-057a-4ca2-ae04-f5d429f470a7","Type":"ContainerDied","Data":"4f9b185b06d1c5750e06f61788ed9836feac27a33048134b8f50b2cda956cb38"} Nov 03 16:45:11.802932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.802545 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m" event={"ID":"96672632-057a-4ca2-ae04-f5d429f470a7","Type":"ContainerDied","Data":"a8f4800f43a9f74be270e96224afc15691eec01fe0309e60e591eb570c9a6c32"} Nov 03 16:45:11.802932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.802581 4947 scope.go:122] "RemoveContainer" containerID="4f9b185b06d1c5750e06f61788ed9836feac27a33048134b8f50b2cda956cb38" Nov 03 16:45:11.804049 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.804028 4947 generic.go:356] "Generic (PLEG): container finished" podID="1548c8a6-c110-4168-9568-5adcc07df255" containerID="fe3693c109056f38f582e883703a09beeb5548a0b753704b738c84f703ff19ac" exitCode=0 Nov 03 16:45:11.804136 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.804059 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8" event={"ID":"1548c8a6-c110-4168-9568-5adcc07df255","Type":"ContainerDied","Data":"fe3693c109056f38f582e883703a09beeb5548a0b753704b738c84f703ff19ac"} Nov 03 16:45:11.804136 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.804088 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8" Nov 03 16:45:11.804136 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.804101 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8" event={"ID":"1548c8a6-c110-4168-9568-5adcc07df255","Type":"ContainerDied","Data":"d387bc8571ce0ad7bba1985aa1fbcfe1879157ba7673322210ec3cc76e877992"} Nov 03 16:45:11.813428 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.813404 4947 scope.go:122] "RemoveContainer" containerID="4f9b185b06d1c5750e06f61788ed9836feac27a33048134b8f50b2cda956cb38" Nov 03 16:45:11.813826 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:45:11.813799 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f9b185b06d1c5750e06f61788ed9836feac27a33048134b8f50b2cda956cb38\": container with ID starting with 4f9b185b06d1c5750e06f61788ed9836feac27a33048134b8f50b2cda956cb38 not found: ID does not exist" containerID="4f9b185b06d1c5750e06f61788ed9836feac27a33048134b8f50b2cda956cb38" Nov 03 16:45:11.813920 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.813833 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f9b185b06d1c5750e06f61788ed9836feac27a33048134b8f50b2cda956cb38"} err="failed to get container status \"4f9b185b06d1c5750e06f61788ed9836feac27a33048134b8f50b2cda956cb38\": rpc error: code = NotFound desc = could not find container \"4f9b185b06d1c5750e06f61788ed9836feac27a33048134b8f50b2cda956cb38\": container with ID starting with 4f9b185b06d1c5750e06f61788ed9836feac27a33048134b8f50b2cda956cb38 not found: ID does not exist" Nov 03 16:45:11.813920 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.813851 4947 scope.go:122] "RemoveContainer" containerID="fe3693c109056f38f582e883703a09beeb5548a0b753704b738c84f703ff19ac" Nov 03 16:45:11.824473 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.824454 4947 scope.go:122] "RemoveContainer" containerID="fe3693c109056f38f582e883703a09beeb5548a0b753704b738c84f703ff19ac" Nov 03 16:45:11.824814 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:45:11.824792 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe3693c109056f38f582e883703a09beeb5548a0b753704b738c84f703ff19ac\": container with ID starting with fe3693c109056f38f582e883703a09beeb5548a0b753704b738c84f703ff19ac not found: ID does not exist" containerID="fe3693c109056f38f582e883703a09beeb5548a0b753704b738c84f703ff19ac" Nov 03 16:45:11.824906 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.824821 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe3693c109056f38f582e883703a09beeb5548a0b753704b738c84f703ff19ac"} err="failed to get container status \"fe3693c109056f38f582e883703a09beeb5548a0b753704b738c84f703ff19ac\": rpc error: code = NotFound desc = could not find container \"fe3693c109056f38f582e883703a09beeb5548a0b753704b738c84f703ff19ac\": container with ID starting with fe3693c109056f38f582e883703a09beeb5548a0b753704b738c84f703ff19ac not found: ID does not exist" Nov 03 16:45:11.861069 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.861035 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m"] Nov 03 16:45:11.872845 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.872808 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bffe15-ff48449d5-t955m"] Nov 03 16:45:11.912968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.912937 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8"] Nov 03 16:45:11.918659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:11.918622 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bffe15-bb845b6dc-d79d8"] Nov 03 16:45:12.191170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.191123 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj"] Nov 03 16:45:12.191956 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.191936 4947 cpu_manager.go:414] "containerMap: removing container" podUID="96672632-057a-4ca2-ae04-f5d429f470a7" containerName="payload-processing" Nov 03 16:45:12.191956 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.191954 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96672632-057a-4ca2-ae04-f5d429f470a7" containerName="payload-processing" Nov 03 16:45:12.192124 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.192037 4947 cpu_manager.go:414] "containerMap: removing container" podUID="1548c8a6-c110-4168-9568-5adcc07df255" containerName="payload-pre-processing" Nov 03 16:45:12.192124 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.192046 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1548c8a6-c110-4168-9568-5adcc07df255" containerName="payload-pre-processing" Nov 03 16:45:12.192240 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.192182 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="1548c8a6-c110-4168-9568-5adcc07df255" containerName="payload-pre-processing" Nov 03 16:45:12.192240 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.192198 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="96672632-057a-4ca2-ae04-f5d429f470a7" containerName="payload-processing" Nov 03 16:45:12.198037 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.198012 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" Nov 03 16:45:12.224210 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.222936 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj"] Nov 03 16:45:12.233058 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.232976 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2cb1bb-serving-cert\"" Nov 03 16:45:12.320475 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.320432 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08739cb-9aea-48c7-ab97-6d57c629880c-metrics-certs\") pod \"maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj\" (UID: \"e08739cb-9aea-48c7-ab97-6d57c629880c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" Nov 03 16:45:12.320668 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.320497 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e08739cb-9aea-48c7-ab97-6d57c629880c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj\" (UID: \"e08739cb-9aea-48c7-ab97-6d57c629880c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" Nov 03 16:45:12.320668 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.320580 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xwnr\" (UniqueName: \"kubernetes.io/projected/e08739cb-9aea-48c7-ab97-6d57c629880c-kube-api-access-6xwnr\") pod \"maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj\" (UID: \"e08739cb-9aea-48c7-ab97-6d57c629880c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" Nov 03 16:45:12.320865 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.320691 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08739cb-9aea-48c7-ab97-6d57c629880c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj\" (UID: \"e08739cb-9aea-48c7-ab97-6d57c629880c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" Nov 03 16:45:12.426722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.426489 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08739cb-9aea-48c7-ab97-6d57c629880c-metrics-certs\") pod \"maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj\" (UID: \"e08739cb-9aea-48c7-ab97-6d57c629880c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" Nov 03 16:45:12.426722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.426587 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e08739cb-9aea-48c7-ab97-6d57c629880c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj\" (UID: \"e08739cb-9aea-48c7-ab97-6d57c629880c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" Nov 03 16:45:12.429729 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.428582 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xwnr\" (UniqueName: \"kubernetes.io/projected/e08739cb-9aea-48c7-ab97-6d57c629880c-kube-api-access-6xwnr\") pod \"maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj\" (UID: \"e08739cb-9aea-48c7-ab97-6d57c629880c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" Nov 03 16:45:12.429729 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.428822 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08739cb-9aea-48c7-ab97-6d57c629880c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj\" (UID: \"e08739cb-9aea-48c7-ab97-6d57c629880c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" Nov 03 16:45:12.433733 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.433645 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08739cb-9aea-48c7-ab97-6d57c629880c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj\" (UID: \"e08739cb-9aea-48c7-ab97-6d57c629880c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" Nov 03 16:45:12.449807 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.442232 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08739cb-9aea-48c7-ab97-6d57c629880c-metrics-certs\") pod \"maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj\" (UID: \"e08739cb-9aea-48c7-ab97-6d57c629880c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" Nov 03 16:45:12.449807 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.442804 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e08739cb-9aea-48c7-ab97-6d57c629880c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj\" (UID: \"e08739cb-9aea-48c7-ab97-6d57c629880c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" Nov 03 16:45:12.474763 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.474436 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xwnr\" (UniqueName: \"kubernetes.io/projected/e08739cb-9aea-48c7-ab97-6d57c629880c-kube-api-access-6xwnr\") pod \"maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj\" (UID: \"e08739cb-9aea-48c7-ab97-6d57c629880c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" Nov 03 16:45:12.529730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.528996 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" Nov 03 16:45:12.582850 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.581513 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1548c8a6-c110-4168-9568-5adcc07df255" path="/var/lib/kubelet/pods/1548c8a6-c110-4168-9568-5adcc07df255/volumes" Nov 03 16:45:12.582850 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.582377 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96672632-057a-4ca2-ae04-f5d429f470a7" path="/var/lib/kubelet/pods/96672632-057a-4ca2-ae04-f5d429f470a7/volumes" Nov 03 16:45:12.894464 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:45:12.894431 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08739cb_9aea_48c7_ab97_6d57c629880c.slice/crio-af8429d15f03a5a837a19c443d5651714834024f19220830b83fe9e5509653cf WatchSource:0}: Error finding container af8429d15f03a5a837a19c443d5651714834024f19220830b83fe9e5509653cf: Status 404 returned error can't find the container with id af8429d15f03a5a837a19c443d5651714834024f19220830b83fe9e5509653cf Nov 03 16:45:12.895824 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:12.895797 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj"] Nov 03 16:45:13.215129 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.215106 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j"] Nov 03 16:45:13.220996 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.220967 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.280726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.278589 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-53bd50-openshift-default-dockercfg-xk9ms\"" Nov 03 16:45:13.280726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.278735 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j"] Nov 03 16:45:13.368784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.358824 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd5zn\" (UniqueName: \"kubernetes.io/projected/70c4edfb-34cc-41d4-9187-c29e1b0a9609-kube-api-access-sd5zn\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.368784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.358907 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-envoy\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.368784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.358971 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-workload-certs\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.368784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.359021 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-token\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.368784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.359071 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-data\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.368784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.359095 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-podinfo\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.368784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.359159 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istiod-ca-cert\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.368784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.359309 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-credential-socket\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.368784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.359360 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-workload-socket\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.504730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.496123 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-envoy\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.545724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.535887 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-workload-certs\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.545724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.536183 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-token\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.545724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.536382 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-data\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.545724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.536574 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-podinfo\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.545724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.536837 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istiod-ca-cert\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.545724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.537432 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-credential-socket\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.545724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.538729 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-workload-socket\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.545724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.539298 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd5zn\" (UniqueName: \"kubernetes.io/projected/70c4edfb-34cc-41d4-9187-c29e1b0a9609-kube-api-access-sd5zn\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.545724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.542961 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-credential-socket\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.545724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.544525 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istiod-ca-cert\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.545724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.544750 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-workload-certs\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.545724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.545519 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-data\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.545724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.545564 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-workload-socket\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.560776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.559775 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-podinfo\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.714729 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.713960 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-envoy\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.722906 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.715354 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-token\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.741077 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.730652 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h" podUID="3a41205a-a408-4718-8a07-eb1870d0e91f" containerName="payload-processing" containerID="cri-o://abfb29ccdb43c0ee9813e885d1564d898f690df2d7978ff3b7599f230ba79899" gracePeriod=25 Nov 03 16:45:13.741077 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.732119 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd5zn\" (UniqueName: \"kubernetes.io/projected/70c4edfb-34cc-41d4-9187-c29e1b0a9609-kube-api-access-sd5zn\") pod \"e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.775784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.775738 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x" podUID="d4b24b1f-3cbd-435d-b59d-c3214a86c39e" containerName="payload-pre-processing" containerID="cri-o://37dbb257c5bd1ce61ff9ac809eb6b78a66fd5dd01ad4221d89f6d0e825818609" gracePeriod=25 Nov 03 16:45:13.822559 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.822516 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" event={"ID":"e08739cb-9aea-48c7-ab97-6d57c629880c","Type":"ContainerStarted","Data":"a1713c5ed697f35e9e06d762ec9c5cd84194630dc31de2bfc0ead4a917c5981f"} Nov 03 16:45:13.822750 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.822583 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" event={"ID":"e08739cb-9aea-48c7-ab97-6d57c629880c","Type":"ContainerStarted","Data":"af8429d15f03a5a837a19c443d5651714834024f19220830b83fe9e5509653cf"} Nov 03 16:45:13.822750 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.822615 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" Nov 03 16:45:13.855056 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.855020 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:13.875844 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:13.875795 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" podStartSLOduration=1.5608242140000002 podStartE2EDuration="1.875781206s" totalImagesPullingTime="314.956992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:13.872989588 +0000 UTC m=+1857.461989560" watchObservedRunningTime="2026-11-03 16:45:13.875781206 +0000 UTC m=+1857.464781174" Nov 03 16:45:14.218257 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:45:14.218223 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c4edfb_34cc_41d4_9187_c29e1b0a9609.slice/crio-44476f7c0332b66d64eaa9c24716c68e4eeca1d9743bc204c88b97baa4183432 WatchSource:0}: Error finding container 44476f7c0332b66d64eaa9c24716c68e4eeca1d9743bc204c88b97baa4183432: Status 404 returned error can't find the container with id 44476f7c0332b66d64eaa9c24716c68e4eeca1d9743bc204c88b97baa4183432 Nov 03 16:45:14.219365 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.219333 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j"] Nov 03 16:45:14.221818 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.221778 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:45:14.221911 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.221875 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:45:14.221953 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.221938 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:45:14.299254 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.299224 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h" Nov 03 16:45:14.302816 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.302792 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x" Nov 03 16:45:14.375087 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.374986 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b24b1f-3cbd-435d-b59d-c3214a86c39e-kube-api-access-q67jv\" (UniqueName: \"kubernetes.io/projected/d4b24b1f-3cbd-435d-b59d-c3214a86c39e-kube-api-access-q67jv\") pod \"d4b24b1f-3cbd-435d-b59d-c3214a86c39e\" (UID: \"d4b24b1f-3cbd-435d-b59d-c3214a86c39e\") " Nov 03 16:45:14.375269 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.375112 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a41205a-a408-4718-8a07-eb1870d0e91f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a41205a-a408-4718-8a07-eb1870d0e91f-plugins-config-volume\") pod \"3a41205a-a408-4718-8a07-eb1870d0e91f\" (UID: \"3a41205a-a408-4718-8a07-eb1870d0e91f\") " Nov 03 16:45:14.375269 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.375247 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a41205a-a408-4718-8a07-eb1870d0e91f-kube-api-access-46cx9\" (UniqueName: \"kubernetes.io/projected/3a41205a-a408-4718-8a07-eb1870d0e91f-kube-api-access-46cx9\") pod \"3a41205a-a408-4718-8a07-eb1870d0e91f\" (UID: \"3a41205a-a408-4718-8a07-eb1870d0e91f\") " Nov 03 16:45:14.375445 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.375425 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4b24b1f-3cbd-435d-b59d-c3214a86c39e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4b24b1f-3cbd-435d-b59d-c3214a86c39e-plugins-config-volume\") pod \"d4b24b1f-3cbd-435d-b59d-c3214a86c39e\" (UID: \"d4b24b1f-3cbd-435d-b59d-c3214a86c39e\") " Nov 03 16:45:14.375544 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.375522 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a41205a-a408-4718-8a07-eb1870d0e91f-plugins-config-volume" pod "3a41205a-a408-4718-8a07-eb1870d0e91f" (UID: "3a41205a-a408-4718-8a07-eb1870d0e91f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:14.375846 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.375821 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b24b1f-3cbd-435d-b59d-c3214a86c39e-plugins-config-volume" pod "d4b24b1f-3cbd-435d-b59d-c3214a86c39e" (UID: "d4b24b1f-3cbd-435d-b59d-c3214a86c39e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:14.376023 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.376006 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4b24b1f-3cbd-435d-b59d-c3214a86c39e-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:14.376088 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.376026 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a41205a-a408-4718-8a07-eb1870d0e91f-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:14.377831 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.377806 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a41205a-a408-4718-8a07-eb1870d0e91f-kube-api-access-46cx9" pod "3a41205a-a408-4718-8a07-eb1870d0e91f" (UID: "3a41205a-a408-4718-8a07-eb1870d0e91f"). InnerVolumeSpecName "kube-api-access-46cx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:14.378095 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.378076 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b24b1f-3cbd-435d-b59d-c3214a86c39e-kube-api-access-q67jv" pod "d4b24b1f-3cbd-435d-b59d-c3214a86c39e" (UID: "d4b24b1f-3cbd-435d-b59d-c3214a86c39e"). InnerVolumeSpecName "kube-api-access-q67jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:14.493787 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.477285 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46cx9\" (UniqueName: \"kubernetes.io/projected/3a41205a-a408-4718-8a07-eb1870d0e91f-kube-api-access-46cx9\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:14.493787 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.477326 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q67jv\" (UniqueName: \"kubernetes.io/projected/d4b24b1f-3cbd-435d-b59d-c3214a86c39e-kube-api-access-q67jv\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:14.831251 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.831215 4947 generic.go:356] "Generic (PLEG): container finished" podID="d4b24b1f-3cbd-435d-b59d-c3214a86c39e" containerID="37dbb257c5bd1ce61ff9ac809eb6b78a66fd5dd01ad4221d89f6d0e825818609" exitCode=0 Nov 03 16:45:14.831465 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.831262 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x" Nov 03 16:45:14.831465 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.831305 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x" event={"ID":"d4b24b1f-3cbd-435d-b59d-c3214a86c39e","Type":"ContainerDied","Data":"37dbb257c5bd1ce61ff9ac809eb6b78a66fd5dd01ad4221d89f6d0e825818609"} Nov 03 16:45:14.831465 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.831359 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x" event={"ID":"d4b24b1f-3cbd-435d-b59d-c3214a86c39e","Type":"ContainerDied","Data":"c47ca6925fe8d008f28cb4d5a4b1dd82842f190f8bbf87993c3ee70ce6833eba"} Nov 03 16:45:14.831465 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.831389 4947 scope.go:122] "RemoveContainer" containerID="37dbb257c5bd1ce61ff9ac809eb6b78a66fd5dd01ad4221d89f6d0e825818609" Nov 03 16:45:14.833537 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.833515 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" event={"ID":"70c4edfb-34cc-41d4-9187-c29e1b0a9609","Type":"ContainerStarted","Data":"3b838ee3794597597e92dd792fa17ae1a73508e706404350328423a2e077100b"} Nov 03 16:45:14.833678 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.833562 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" event={"ID":"70c4edfb-34cc-41d4-9187-c29e1b0a9609","Type":"ContainerStarted","Data":"44476f7c0332b66d64eaa9c24716c68e4eeca1d9743bc204c88b97baa4183432"} Nov 03 16:45:14.835563 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.835539 4947 generic.go:356] "Generic (PLEG): container finished" podID="3a41205a-a408-4718-8a07-eb1870d0e91f" containerID="abfb29ccdb43c0ee9813e885d1564d898f690df2d7978ff3b7599f230ba79899" exitCode=0 Nov 03 16:45:14.835682 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.835573 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h" event={"ID":"3a41205a-a408-4718-8a07-eb1870d0e91f","Type":"ContainerDied","Data":"abfb29ccdb43c0ee9813e885d1564d898f690df2d7978ff3b7599f230ba79899"} Nov 03 16:45:14.835682 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.835589 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h" Nov 03 16:45:14.835682 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.835622 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h" event={"ID":"3a41205a-a408-4718-8a07-eb1870d0e91f","Type":"ContainerDied","Data":"af3c0552251b6868fbb2d275e9158b74adfb2e1d5dbad68a03ffb0e7a328640d"} Nov 03 16:45:14.844868 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.844846 4947 scope.go:122] "RemoveContainer" containerID="37dbb257c5bd1ce61ff9ac809eb6b78a66fd5dd01ad4221d89f6d0e825818609" Nov 03 16:45:14.845249 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:45:14.845223 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37dbb257c5bd1ce61ff9ac809eb6b78a66fd5dd01ad4221d89f6d0e825818609\": container with ID starting with 37dbb257c5bd1ce61ff9ac809eb6b78a66fd5dd01ad4221d89f6d0e825818609 not found: ID does not exist" containerID="37dbb257c5bd1ce61ff9ac809eb6b78a66fd5dd01ad4221d89f6d0e825818609" Nov 03 16:45:14.845326 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.845266 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37dbb257c5bd1ce61ff9ac809eb6b78a66fd5dd01ad4221d89f6d0e825818609"} err="failed to get container status \"37dbb257c5bd1ce61ff9ac809eb6b78a66fd5dd01ad4221d89f6d0e825818609\": rpc error: code = NotFound desc = could not find container \"37dbb257c5bd1ce61ff9ac809eb6b78a66fd5dd01ad4221d89f6d0e825818609\": container with ID starting with 37dbb257c5bd1ce61ff9ac809eb6b78a66fd5dd01ad4221d89f6d0e825818609 not found: ID does not exist" Nov 03 16:45:14.845326 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.845286 4947 scope.go:122] "RemoveContainer" containerID="abfb29ccdb43c0ee9813e885d1564d898f690df2d7978ff3b7599f230ba79899" Nov 03 16:45:14.856077 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.855900 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:14.856152 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.856123 4947 scope.go:122] "RemoveContainer" containerID="abfb29ccdb43c0ee9813e885d1564d898f690df2d7978ff3b7599f230ba79899" Nov 03 16:45:14.856530 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:45:14.856508 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abfb29ccdb43c0ee9813e885d1564d898f690df2d7978ff3b7599f230ba79899\": container with ID starting with abfb29ccdb43c0ee9813e885d1564d898f690df2d7978ff3b7599f230ba79899 not found: ID does not exist" containerID="abfb29ccdb43c0ee9813e885d1564d898f690df2d7978ff3b7599f230ba79899" Nov 03 16:45:14.856602 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.856539 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abfb29ccdb43c0ee9813e885d1564d898f690df2d7978ff3b7599f230ba79899"} err="failed to get container status \"abfb29ccdb43c0ee9813e885d1564d898f690df2d7978ff3b7599f230ba79899\": rpc error: code = NotFound desc = could not find container \"abfb29ccdb43c0ee9813e885d1564d898f690df2d7978ff3b7599f230ba79899\": container with ID starting with abfb29ccdb43c0ee9813e885d1564d898f690df2d7978ff3b7599f230ba79899 not found: ID does not exist" Nov 03 16:45:14.901656 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.901595 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" podStartSLOduration=1.901580913 podStartE2EDuration="1.901580913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:14.901176186 +0000 UTC m=+1858.490176155" watchObservedRunningTime="2026-11-03 16:45:14.901580913 +0000 UTC m=+1858.490580876" Nov 03 16:45:14.941489 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.941453 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h"] Nov 03 16:45:14.950028 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.949994 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8650a6e8-5f9b864889-zsp7h"] Nov 03 16:45:14.987188 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.987089 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x"] Nov 03 16:45:14.991091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:14.991064 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8650a6e8-5899ddc679-dwr5x"] Nov 03 16:45:15.861613 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:15.861582 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:16.568116 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:16.568083 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a41205a-a408-4718-8a07-eb1870d0e91f" path="/var/lib/kubelet/pods/3a41205a-a408-4718-8a07-eb1870d0e91f/volumes" Nov 03 16:45:16.568557 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:16.568543 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b24b1f-3cbd-435d-b59d-c3214a86c39e" path="/var/lib/kubelet/pods/d4b24b1f-3cbd-435d-b59d-c3214a86c39e/volumes" Nov 03 16:45:16.853233 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:16.853144 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:16.854290 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:16.854260 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:45:19.265583 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.265548 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4"] Nov 03 16:45:19.266291 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.266274 4947 cpu_manager.go:414] "containerMap: removing container" podUID="3a41205a-a408-4718-8a07-eb1870d0e91f" containerName="payload-processing" Nov 03 16:45:19.266340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.266293 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a41205a-a408-4718-8a07-eb1870d0e91f" containerName="payload-processing" Nov 03 16:45:19.266340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.266306 4947 cpu_manager.go:414] "containerMap: removing container" podUID="d4b24b1f-3cbd-435d-b59d-c3214a86c39e" containerName="payload-pre-processing" Nov 03 16:45:19.266340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.266312 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b24b1f-3cbd-435d-b59d-c3214a86c39e" containerName="payload-pre-processing" Nov 03 16:45:19.266455 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.266437 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a41205a-a408-4718-8a07-eb1870d0e91f" containerName="payload-processing" Nov 03 16:45:19.266455 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.266445 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b24b1f-3cbd-435d-b59d-c3214a86c39e" containerName="payload-pre-processing" Nov 03 16:45:19.272582 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.272554 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4" Nov 03 16:45:19.298762 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.297412 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-53bd50\"" Nov 03 16:45:19.302002 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.301963 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4"] Nov 03 16:45:19.310676 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.310644 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-53bd50-dockercfg-v5brf\"" Nov 03 16:45:19.381722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.368678 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7zb8\" (UniqueName: \"kubernetes.io/projected/c1dd2f69-49df-4214-bb00-9f2cfe7073df-kube-api-access-g7zb8\") pod \"payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4\" (UID: \"c1dd2f69-49df-4214-bb00-9f2cfe7073df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4" Nov 03 16:45:19.381722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.369046 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1dd2f69-49df-4214-bb00-9f2cfe7073df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4\" (UID: \"c1dd2f69-49df-4214-bb00-9f2cfe7073df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4" Nov 03 16:45:19.504733 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.496959 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1dd2f69-49df-4214-bb00-9f2cfe7073df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4\" (UID: \"c1dd2f69-49df-4214-bb00-9f2cfe7073df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4" Nov 03 16:45:19.504733 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.497147 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7zb8\" (UniqueName: \"kubernetes.io/projected/c1dd2f69-49df-4214-bb00-9f2cfe7073df-kube-api-access-g7zb8\") pod \"payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4\" (UID: \"c1dd2f69-49df-4214-bb00-9f2cfe7073df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4" Nov 03 16:45:19.560776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.559923 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1dd2f69-49df-4214-bb00-9f2cfe7073df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4\" (UID: \"c1dd2f69-49df-4214-bb00-9f2cfe7073df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4" Nov 03 16:45:19.683734 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.661257 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7zb8\" (UniqueName: \"kubernetes.io/projected/c1dd2f69-49df-4214-bb00-9f2cfe7073df-kube-api-access-g7zb8\") pod \"payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4\" (UID: \"c1dd2f69-49df-4214-bb00-9f2cfe7073df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4" Nov 03 16:45:19.842045 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.841965 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" Nov 03 16:45:19.890136 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:19.890102 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4" Nov 03 16:45:20.213199 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:45:20.213152 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1dd2f69_49df_4214_bb00_9f2cfe7073df.slice/crio-0bc50a9f97c5fa7f832e3f083aa4ada7b75d7bb4b800d2d6ff50fc70c77afbae WatchSource:0}: Error finding container 0bc50a9f97c5fa7f832e3f083aa4ada7b75d7bb4b800d2d6ff50fc70c77afbae: Status 404 returned error can't find the container with id 0bc50a9f97c5fa7f832e3f083aa4ada7b75d7bb4b800d2d6ff50fc70c77afbae Nov 03 16:45:20.213740 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:20.213580 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4"] Nov 03 16:45:20.915767 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:20.915719 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4" event={"ID":"c1dd2f69-49df-4214-bb00-9f2cfe7073df","Type":"ContainerStarted","Data":"c928d929de117205884d355f6a37b2dd085f118f4edf07d8953ab93efbd76333"} Nov 03 16:45:20.916291 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:20.915829 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4" Nov 03 16:45:20.916291 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:20.915856 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4" event={"ID":"c1dd2f69-49df-4214-bb00-9f2cfe7073df","Type":"ContainerStarted","Data":"0bc50a9f97c5fa7f832e3f083aa4ada7b75d7bb4b800d2d6ff50fc70c77afbae"} Nov 03 16:45:20.969291 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:20.969234 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4" podStartSLOduration=1.969219008 podStartE2EDuration="1.969219008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:20.968031505 +0000 UTC m=+1864.557031474" watchObservedRunningTime="2026-11-03 16:45:20.969219008 +0000 UTC m=+1864.558218977" Nov 03 16:45:22.507379 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:22.507342 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn"] Nov 03 16:45:22.512068 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:22.512046 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" Nov 03 16:45:22.534392 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:22.534354 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2aa2ad-serving-cert\"" Nov 03 16:45:22.540680 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:22.540642 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn"] Nov 03 16:45:22.646362 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:22.646326 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/176f518a-26d8-45e1-bfd2-a1cdeba229d0-maas-api-tls\") pod \"maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn\" (UID: \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" Nov 03 16:45:22.646569 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:22.646397 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwkgq\" (UniqueName: \"kubernetes.io/projected/176f518a-26d8-45e1-bfd2-a1cdeba229d0-kube-api-access-lwkgq\") pod \"maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn\" (UID: \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" Nov 03 16:45:22.646569 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:22.646490 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/176f518a-26d8-45e1-bfd2-a1cdeba229d0-openshift-service-ca\") pod \"maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn\" (UID: \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" Nov 03 16:45:22.646655 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:22.646613 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/176f518a-26d8-45e1-bfd2-a1cdeba229d0-metrics-certs\") pod \"maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn\" (UID: \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" Nov 03 16:45:22.763148 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:22.763047 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/176f518a-26d8-45e1-bfd2-a1cdeba229d0-openshift-service-ca\") pod \"maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn\" (UID: \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" Nov 03 16:45:22.774347 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:22.765060 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/176f518a-26d8-45e1-bfd2-a1cdeba229d0-openshift-service-ca\") pod \"maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn\" (UID: \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" Nov 03 16:45:22.774656 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:22.774627 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/176f518a-26d8-45e1-bfd2-a1cdeba229d0-metrics-certs\") pod \"maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn\" (UID: \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" Nov 03 16:45:22.775729 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:22.775667 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/176f518a-26d8-45e1-bfd2-a1cdeba229d0-maas-api-tls\") pod \"maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn\" (UID: \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" Nov 03 16:45:22.775871 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:22.775800 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwkgq\" (UniqueName: \"kubernetes.io/projected/176f518a-26d8-45e1-bfd2-a1cdeba229d0-kube-api-access-lwkgq\") pod \"maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn\" (UID: \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" Nov 03 16:45:22.810096 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:22.789445 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/176f518a-26d8-45e1-bfd2-a1cdeba229d0-metrics-certs\") pod \"maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn\" (UID: \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" Nov 03 16:45:22.810096 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:22.804226 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/176f518a-26d8-45e1-bfd2-a1cdeba229d0-maas-api-tls\") pod \"maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn\" (UID: \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" Nov 03 16:45:22.823108 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:22.815036 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwkgq\" (UniqueName: \"kubernetes.io/projected/176f518a-26d8-45e1-bfd2-a1cdeba229d0-kube-api-access-lwkgq\") pod \"maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn\" (UID: \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" Nov 03 16:45:22.845733 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:22.839299 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" Nov 03 16:45:23.211243 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:45:23.211218 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176f518a_26d8_45e1_bfd2_a1cdeba229d0.slice/crio-82a5e9a4a66cf8cb7b81b249021a3b55672dc5a54d75ab66da720f98d2f0e44f WatchSource:0}: Error finding container 82a5e9a4a66cf8cb7b81b249021a3b55672dc5a54d75ab66da720f98d2f0e44f: Status 404 returned error can't find the container with id 82a5e9a4a66cf8cb7b81b249021a3b55672dc5a54d75ab66da720f98d2f0e44f Nov 03 16:45:23.211352 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:23.211295 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn"] Nov 03 16:45:23.936315 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:23.936221 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" event={"ID":"176f518a-26d8-45e1-bfd2-a1cdeba229d0","Type":"ContainerStarted","Data":"603db4c3b190432e2537a65e38de0e8d44cbe38b4db332b2f8c170462cb1e921"} Nov 03 16:45:23.936315 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:23.936301 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" event={"ID":"176f518a-26d8-45e1-bfd2-a1cdeba229d0","Type":"ContainerStarted","Data":"82a5e9a4a66cf8cb7b81b249021a3b55672dc5a54d75ab66da720f98d2f0e44f"} Nov 03 16:45:23.936821 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:23.936450 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" Nov 03 16:45:23.985817 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:23.985758 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" podStartSLOduration=1.651493654 podStartE2EDuration="1.985739806s" totalImagesPullingTime="334.246152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:23.983060736 +0000 UTC m=+1867.572060705" watchObservedRunningTime="2026-11-03 16:45:23.985739806 +0000 UTC m=+1867.574739776" Nov 03 16:45:25.810570 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:25.810537 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr"] Nov 03 16:45:25.811115 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:25.810784 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" podUID="6f7c9da9-3f09-42c2-8b73-3a704acf074d" containerName="maas-api" containerID="cri-o://67114341aca30641363b5c80e59c33f0c69ebc555499d79efce4ae37825aef84" gracePeriod=30 Nov 03 16:45:26.073118 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:26.072861 4947 generic.go:356] "Generic (PLEG): container finished" podID="6f7c9da9-3f09-42c2-8b73-3a704acf074d" containerID="67114341aca30641363b5c80e59c33f0c69ebc555499d79efce4ae37825aef84" exitCode=0 Nov 03 16:45:26.073118 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:26.073023 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" event={"ID":"6f7c9da9-3f09-42c2-8b73-3a704acf074d","Type":"ContainerDied","Data":"67114341aca30641363b5c80e59c33f0c69ebc555499d79efce4ae37825aef84"} Nov 03 16:45:26.961937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:26.961805 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:45:27.104564 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:27.104532 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" event={"ID":"6f7c9da9-3f09-42c2-8b73-3a704acf074d","Type":"ContainerDied","Data":"2b651e1fa3a8f647be3b58c6097ce65042a5bcf8efb81900e1a11aef22e9d353"} Nov 03 16:45:27.104964 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:27.104865 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr" Nov 03 16:45:27.105568 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:27.105551 4947 scope.go:122] "RemoveContainer" containerID="67114341aca30641363b5c80e59c33f0c69ebc555499d79efce4ae37825aef84" Nov 03 16:45:27.174353 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:27.164430 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7c9da9-3f09-42c2-8b73-3a704acf074d-kube-api-access-rc29g\" (UniqueName: \"kubernetes.io/projected/6f7c9da9-3f09-42c2-8b73-3a704acf074d-kube-api-access-rc29g\") pod \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\" (UID: \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\") " Nov 03 16:45:27.174353 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:27.164494 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f7c9da9-3f09-42c2-8b73-3a704acf074d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f7c9da9-3f09-42c2-8b73-3a704acf074d-maas-api-tls\") pod \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\" (UID: \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\") " Nov 03 16:45:27.174353 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:27.164596 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f7c9da9-3f09-42c2-8b73-3a704acf074d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f7c9da9-3f09-42c2-8b73-3a704acf074d-metrics-certs\") pod \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\" (UID: \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\") " Nov 03 16:45:27.174353 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:27.164770 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f7c9da9-3f09-42c2-8b73-3a704acf074d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f7c9da9-3f09-42c2-8b73-3a704acf074d-openshift-service-ca\") pod \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\" (UID: \"6f7c9da9-3f09-42c2-8b73-3a704acf074d\") " Nov 03 16:45:27.197730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:27.193993 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7c9da9-3f09-42c2-8b73-3a704acf074d-openshift-service-ca" pod "6f7c9da9-3f09-42c2-8b73-3a704acf074d" (UID: "6f7c9da9-3f09-42c2-8b73-3a704acf074d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:27.243791 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:27.237086 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7c9da9-3f09-42c2-8b73-3a704acf074d-kube-api-access-rc29g" pod "6f7c9da9-3f09-42c2-8b73-3a704acf074d" (UID: "6f7c9da9-3f09-42c2-8b73-3a704acf074d"). InnerVolumeSpecName "kube-api-access-rc29g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:27.243791 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:27.237385 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f7c9da9-3f09-42c2-8b73-3a704acf074d-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:27.267732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:27.262485 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f7c9da9-3f09-42c2-8b73-3a704acf074d-metrics-certs" pod "6f7c9da9-3f09-42c2-8b73-3a704acf074d" (UID: "6f7c9da9-3f09-42c2-8b73-3a704acf074d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:45:27.267732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:27.262746 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f7c9da9-3f09-42c2-8b73-3a704acf074d-maas-api-tls" pod "6f7c9da9-3f09-42c2-8b73-3a704acf074d" (UID: "6f7c9da9-3f09-42c2-8b73-3a704acf074d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:45:27.354890 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:27.346051 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc29g\" (UniqueName: \"kubernetes.io/projected/6f7c9da9-3f09-42c2-8b73-3a704acf074d-kube-api-access-rc29g\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:27.354890 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:27.346089 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f7c9da9-3f09-42c2-8b73-3a704acf074d-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:27.354890 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:27.346104 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f7c9da9-3f09-42c2-8b73-3a704acf074d-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:27.560465 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:27.560425 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr"] Nov 03 16:45:27.569085 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:27.569054 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8650a6e8-7bc77c6b9-jbvlr"] Nov 03 16:45:27.577611 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:45:27.577568 4947 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7c9da9_3f09_42c2_8b73_3a704acf074d.slice\": RecentStats: unable to find data in memory cache]" Nov 03 16:45:28.566541 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:28.566506 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f7c9da9-3f09-42c2-8b73-3a704acf074d" path="/var/lib/kubelet/pods/6f7c9da9-3f09-42c2-8b73-3a704acf074d/volumes" Nov 03 16:45:29.948818 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:29.948790 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" Nov 03 16:45:30.140157 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:45:30.140114 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:45:31.922369 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:31.922332 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4" Nov 03 16:45:32.542170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:32.542135 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt"] Nov 03 16:45:32.542676 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:32.542640 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" podUID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" containerName="istio-proxy" containerID="cri-o://60f5c148d442bd24a3b74e4ac53374f25160c1a1c2027c53d4383ebce6d9a577" gracePeriod=30 Nov 03 16:45:37.944364 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:37.944247 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb"] Nov 03 16:45:37.945919 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:37.945863 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" podUID="f3de2da9-631c-4ff3-96d3-8cf575990923" containerName="istio-proxy" containerID="cri-o://be798cf05b2c408523f3672273d454f1b1a6726878a74e10e7d9575bb687d4f9" gracePeriod=30 Nov 03 16:45:38.075295 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.074939 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:45:38.134359 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.134320 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-data\") pod \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " Nov 03 16:45:38.145853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.145819 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-token\" (UniqueName: \"kubernetes.io/projected/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-token\") pod \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " Nov 03 16:45:38.153839 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.153803 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-envoy\") pod \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " Nov 03 16:45:38.172770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.154173 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-credential-socket\") pod \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " Nov 03 16:45:38.173158 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.173126 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istiod-ca-cert\") pod \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " Nov 03 16:45:38.255722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.251834 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-workload-certs\") pod \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " Nov 03 16:45:38.255722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.251949 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-podinfo\") pod \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " Nov 03 16:45:38.255722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.252018 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-kube-api-access-75bbk\" (UniqueName: \"kubernetes.io/projected/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-kube-api-access-75bbk\") pod \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " Nov 03 16:45:38.255722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.252119 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-workload-socket\") pod \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\" (UID: \"595b44f5-49e0-4cd7-bd8f-83a72f6f426c\") " Nov 03 16:45:38.270117 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.136033 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-data" pod "595b44f5-49e0-4cd7-bd8f-83a72f6f426c" (UID: "595b44f5-49e0-4cd7-bd8f-83a72f6f426c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:38.270117 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.156107 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-credential-socket" pod "595b44f5-49e0-4cd7-bd8f-83a72f6f426c" (UID: "595b44f5-49e0-4cd7-bd8f-83a72f6f426c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:38.270117 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.173653 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istiod-ca-cert" pod "595b44f5-49e0-4cd7-bd8f-83a72f6f426c" (UID: "595b44f5-49e0-4cd7-bd8f-83a72f6f426c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:38.307750 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.272024 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-podinfo" pod "595b44f5-49e0-4cd7-bd8f-83a72f6f426c" (UID: "595b44f5-49e0-4cd7-bd8f-83a72f6f426c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:45:38.331740 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.327813 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-workload-socket" pod "595b44f5-49e0-4cd7-bd8f-83a72f6f426c" (UID: "595b44f5-49e0-4cd7-bd8f-83a72f6f426c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:38.355646 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.355227 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:38.355646 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.355263 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:38.355646 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.355283 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:38.355646 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.355297 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:38.355646 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.355311 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:38.357168 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.357115 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-kube-api-access-75bbk" pod "595b44f5-49e0-4cd7-bd8f-83a72f6f426c" (UID: "595b44f5-49e0-4cd7-bd8f-83a72f6f426c"). InnerVolumeSpecName "kube-api-access-75bbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:38.360745 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.358878 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-workload-certs" pod "595b44f5-49e0-4cd7-bd8f-83a72f6f426c" (UID: "595b44f5-49e0-4cd7-bd8f-83a72f6f426c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:38.378351 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.378289 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-token" pod "595b44f5-49e0-4cd7-bd8f-83a72f6f426c" (UID: "595b44f5-49e0-4cd7-bd8f-83a72f6f426c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:38.382281 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.382025 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-envoy" pod "595b44f5-49e0-4cd7-bd8f-83a72f6f426c" (UID: "595b44f5-49e0-4cd7-bd8f-83a72f6f426c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:38.453761 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.453663 4947 generic.go:356] "Generic (PLEG): container finished" podID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" containerID="60f5c148d442bd24a3b74e4ac53374f25160c1a1c2027c53d4383ebce6d9a577" exitCode=0 Nov 03 16:45:38.453761 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.453733 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" Nov 03 16:45:38.453761 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.453745 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" event={"ID":"595b44f5-49e0-4cd7-bd8f-83a72f6f426c","Type":"ContainerDied","Data":"60f5c148d442bd24a3b74e4ac53374f25160c1a1c2027c53d4383ebce6d9a577"} Nov 03 16:45:38.454040 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.453814 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt" event={"ID":"595b44f5-49e0-4cd7-bd8f-83a72f6f426c","Type":"ContainerDied","Data":"97f0f65904aa8d12ad5876a1d83a739625e6e219e4056ef31f81a35dbb1da99e"} Nov 03 16:45:38.454040 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.453843 4947 scope.go:122] "RemoveContainer" containerID="60f5c148d442bd24a3b74e4ac53374f25160c1a1c2027c53d4383ebce6d9a577" Nov 03 16:45:38.456395 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.456368 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:38.456395 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.456394 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:38.456592 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.456403 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:38.456592 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.456413 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75bbk\" (UniqueName: \"kubernetes.io/projected/595b44f5-49e0-4cd7-bd8f-83a72f6f426c-kube-api-access-75bbk\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:38.465926 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.465902 4947 scope.go:122] "RemoveContainer" containerID="9942e727094dd7306b774fa7a1b12ae4965247ec53b3428af1234165a67c53b2" Nov 03 16:45:38.475997 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.475970 4947 scope.go:122] "RemoveContainer" containerID="60f5c148d442bd24a3b74e4ac53374f25160c1a1c2027c53d4383ebce6d9a577" Nov 03 16:45:38.476644 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:45:38.476316 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f5c148d442bd24a3b74e4ac53374f25160c1a1c2027c53d4383ebce6d9a577\": container with ID starting with 60f5c148d442bd24a3b74e4ac53374f25160c1a1c2027c53d4383ebce6d9a577 not found: ID does not exist" containerID="60f5c148d442bd24a3b74e4ac53374f25160c1a1c2027c53d4383ebce6d9a577" Nov 03 16:45:38.476644 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.476347 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f5c148d442bd24a3b74e4ac53374f25160c1a1c2027c53d4383ebce6d9a577"} err="failed to get container status \"60f5c148d442bd24a3b74e4ac53374f25160c1a1c2027c53d4383ebce6d9a577\": rpc error: code = NotFound desc = could not find container \"60f5c148d442bd24a3b74e4ac53374f25160c1a1c2027c53d4383ebce6d9a577\": container with ID starting with 60f5c148d442bd24a3b74e4ac53374f25160c1a1c2027c53d4383ebce6d9a577 not found: ID does not exist" Nov 03 16:45:38.476644 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.476366 4947 scope.go:122] "RemoveContainer" containerID="9942e727094dd7306b774fa7a1b12ae4965247ec53b3428af1234165a67c53b2" Nov 03 16:45:38.476644 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:45:38.476630 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9942e727094dd7306b774fa7a1b12ae4965247ec53b3428af1234165a67c53b2\": container with ID starting with 9942e727094dd7306b774fa7a1b12ae4965247ec53b3428af1234165a67c53b2 not found: ID does not exist" containerID="9942e727094dd7306b774fa7a1b12ae4965247ec53b3428af1234165a67c53b2" Nov 03 16:45:38.476922 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.476645 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9942e727094dd7306b774fa7a1b12ae4965247ec53b3428af1234165a67c53b2"} err="failed to get container status \"9942e727094dd7306b774fa7a1b12ae4965247ec53b3428af1234165a67c53b2\": rpc error: code = NotFound desc = could not find container \"9942e727094dd7306b774fa7a1b12ae4965247ec53b3428af1234165a67c53b2\": container with ID starting with 9942e727094dd7306b774fa7a1b12ae4965247ec53b3428af1234165a67c53b2 not found: ID does not exist" Nov 03 16:45:38.537645 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.537564 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt"] Nov 03 16:45:38.550460 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.550425 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bffe15-openshift-default-5bdcd78699-blbkt"] Nov 03 16:45:38.567315 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:38.567281 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" path="/var/lib/kubelet/pods/595b44f5-49e0-4cd7-bd8f-83a72f6f426c/volumes" Nov 03 16:45:39.364360 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.364321 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn"] Nov 03 16:45:39.365393 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.365368 4947 cpu_manager.go:414] "containerMap: removing container" podUID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" containerName="istio-proxy" Nov 03 16:45:39.365492 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.365396 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" containerName="istio-proxy" Nov 03 16:45:39.365492 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.365451 4947 cpu_manager.go:414] "containerMap: removing container" podUID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" containerName="istio-proxy" Nov 03 16:45:39.365492 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.365459 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" containerName="istio-proxy" Nov 03 16:45:39.365591 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.365503 4947 cpu_manager.go:414] "containerMap: removing container" podUID="6f7c9da9-3f09-42c2-8b73-3a704acf074d" containerName="maas-api" Nov 03 16:45:39.365591 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.365511 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7c9da9-3f09-42c2-8b73-3a704acf074d" containerName="maas-api" Nov 03 16:45:39.365708 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.365677 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f7c9da9-3f09-42c2-8b73-3a704acf074d" containerName="maas-api" Nov 03 16:45:39.365782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.365726 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" containerName="istio-proxy" Nov 03 16:45:39.365782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.365743 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="595b44f5-49e0-4cd7-bd8f-83a72f6f426c" containerName="istio-proxy" Nov 03 16:45:39.371648 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.371626 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.400655 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.400624 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn"] Nov 03 16:45:39.405560 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.405530 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-98e3b9-openshift-default-dockercfg-dwk5p\"" Nov 03 16:45:39.490229 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.490189 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/290e2d53-a6ba-4406-af23-1f2304d80983-istio-podinfo\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.490409 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.490261 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4q5g\" (UniqueName: \"kubernetes.io/projected/290e2d53-a6ba-4406-af23-1f2304d80983-kube-api-access-t4q5g\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.490409 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.490377 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-istio-envoy\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.490409 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.490403 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-credential-socket\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.490545 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.490436 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-workload-certs\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.490545 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.490530 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/290e2d53-a6ba-4406-af23-1f2304d80983-istiod-ca-cert\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.490651 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.490636 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/290e2d53-a6ba-4406-af23-1f2304d80983-istio-token\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.490784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.490766 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-workload-socket\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.490864 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.490813 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-istio-data\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.592585 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.592483 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-workload-socket\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.592585 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.592549 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-istio-data\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.592886 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.592625 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/290e2d53-a6ba-4406-af23-1f2304d80983-istio-podinfo\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.592886 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.592675 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4q5g\" (UniqueName: \"kubernetes.io/projected/290e2d53-a6ba-4406-af23-1f2304d80983-kube-api-access-t4q5g\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.592886 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.592775 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-istio-envoy\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.592886 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.592798 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-credential-socket\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.592886 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.592843 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-workload-certs\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.592886 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.592882 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/290e2d53-a6ba-4406-af23-1f2304d80983-istiod-ca-cert\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.593202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.593000 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/290e2d53-a6ba-4406-af23-1f2304d80983-istio-token\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.593202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.593069 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-workload-socket\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.593202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.593101 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-istio-data\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.593503 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.593466 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-credential-socket\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.593503 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.593466 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-workload-certs\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.594201 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.594174 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/290e2d53-a6ba-4406-af23-1f2304d80983-istiod-ca-cert\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.595601 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.595574 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-istio-envoy\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.595890 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.595870 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/290e2d53-a6ba-4406-af23-1f2304d80983-istio-podinfo\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.653730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.652954 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/290e2d53-a6ba-4406-af23-1f2304d80983-istio-token\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.657851 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.657784 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4q5g\" (UniqueName: \"kubernetes.io/projected/290e2d53-a6ba-4406-af23-1f2304d80983-kube-api-access-t4q5g\") pod \"e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:39.699528 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:39.699029 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:40.239540 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:40.239504 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn"] Nov 03 16:45:40.246113 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:40.246060 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:45:40.246304 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:40.246272 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:45:40.246536 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:40.246511 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:45:40.472484 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:40.472451 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" event={"ID":"290e2d53-a6ba-4406-af23-1f2304d80983","Type":"ContainerStarted","Data":"b200513447e4cd77122b250138f46c3f77883214480ceb9f50bc6b38715cd34d"} Nov 03 16:45:40.473069 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:40.473034 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" event={"ID":"290e2d53-a6ba-4406-af23-1f2304d80983","Type":"ContainerStarted","Data":"b17b6366d1162731e4dd9835b87ffb4edb4d10847b31a56cdec240596fe1a58c"} Nov 03 16:45:40.523062 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:40.522999 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" podStartSLOduration=1.5229840559999999 podStartE2EDuration="1.522984056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:40.521260146 +0000 UTC m=+1884.110260115" watchObservedRunningTime="2026-11-03 16:45:40.522984056 +0000 UTC m=+1884.111984025" Nov 03 16:45:40.699562 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:40.699530 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:41.608352 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:41.608312 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n"] Nov 03 16:45:41.613250 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:41.613224 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:45:41.642969 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:41.642929 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-53bd50-serving-cert\"" Nov 03 16:45:41.651755 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:41.650656 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n"] Nov 03 16:45:41.705485 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:41.705448 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:41.711395 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:41.711361 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cedd499-42b4-49bc-937b-e86a540b9094-metrics-certs\") pod \"maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n\" (UID: \"1cedd499-42b4-49bc-937b-e86a540b9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:45:41.711576 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:41.711433 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtmss\" (UniqueName: \"kubernetes.io/projected/1cedd499-42b4-49bc-937b-e86a540b9094-kube-api-access-wtmss\") pod \"maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n\" (UID: \"1cedd499-42b4-49bc-937b-e86a540b9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:45:41.711658 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:41.711567 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cedd499-42b4-49bc-937b-e86a540b9094-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n\" (UID: \"1cedd499-42b4-49bc-937b-e86a540b9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:45:41.711658 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:41.711642 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cedd499-42b4-49bc-937b-e86a540b9094-maas-api-tls\") pod \"maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n\" (UID: \"1cedd499-42b4-49bc-937b-e86a540b9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:45:41.824899 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:41.824845 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cedd499-42b4-49bc-937b-e86a540b9094-metrics-certs\") pod \"maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n\" (UID: \"1cedd499-42b4-49bc-937b-e86a540b9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:45:41.825077 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:41.824923 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtmss\" (UniqueName: \"kubernetes.io/projected/1cedd499-42b4-49bc-937b-e86a540b9094-kube-api-access-wtmss\") pod \"maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n\" (UID: \"1cedd499-42b4-49bc-937b-e86a540b9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:45:41.825077 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:41.824998 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cedd499-42b4-49bc-937b-e86a540b9094-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n\" (UID: \"1cedd499-42b4-49bc-937b-e86a540b9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:45:41.825200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:41.825072 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cedd499-42b4-49bc-937b-e86a540b9094-maas-api-tls\") pod \"maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n\" (UID: \"1cedd499-42b4-49bc-937b-e86a540b9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:45:41.825402 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:45:41.825333 4947 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-serving-cert: secret "maas-api-e2e-ipp-b-53bd50-serving-cert" not found Nov 03 16:45:41.825482 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:45:41.825405 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cedd499-42b4-49bc-937b-e86a540b9094-maas-api-tls podName:1cedd499-42b4-49bc-937b-e86a540b9094 nodeName:}" failed. No retries permitted until 2026-11-03 16:45:42.325383944 +0000 UTC m=+1885.914383907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1cedd499-42b4-49bc-937b-e86a540b9094-maas-api-tls") pod "maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" (UID: "1cedd499-42b4-49bc-937b-e86a540b9094") : secret "maas-api-e2e-ipp-b-53bd50-serving-cert" not found Nov 03 16:45:41.847741 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:41.836112 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cedd499-42b4-49bc-937b-e86a540b9094-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n\" (UID: \"1cedd499-42b4-49bc-937b-e86a540b9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:45:41.857421 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:41.856022 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cedd499-42b4-49bc-937b-e86a540b9094-metrics-certs\") pod \"maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n\" (UID: \"1cedd499-42b4-49bc-937b-e86a540b9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:45:41.873826 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:41.865215 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtmss\" (UniqueName: \"kubernetes.io/projected/1cedd499-42b4-49bc-937b-e86a540b9094-kube-api-access-wtmss\") pod \"maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n\" (UID: \"1cedd499-42b4-49bc-937b-e86a540b9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:45:42.334525 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:42.334487 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cedd499-42b4-49bc-937b-e86a540b9094-maas-api-tls\") pod \"maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n\" (UID: \"1cedd499-42b4-49bc-937b-e86a540b9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:45:42.337236 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:42.337199 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cedd499-42b4-49bc-937b-e86a540b9094-maas-api-tls\") pod \"maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n\" (UID: \"1cedd499-42b4-49bc-937b-e86a540b9094\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:45:42.547659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:42.547620 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:42.552214 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:42.549667 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:45:42.553067 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:42.552821 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:45:42.941865 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:42.941752 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n"] Nov 03 16:45:43.750251 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.750219 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:45:43.755443 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.755408 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4"] Nov 03 16:45:43.758202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.758152 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" podUID="e7f1da65-4f9b-4512-b1fb-66c719d89abb" containerName="maas-api" containerID="cri-o://21b8362638fd1ba2174ae72765c07f0db457264f4f6e37f5c4767ba9830e7210" gracePeriod=30 Nov 03 16:45:43.801822 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.801786 4947 generic.go:356] "Generic (PLEG): container finished" podID="f3de2da9-631c-4ff3-96d3-8cf575990923" containerID="be798cf05b2c408523f3672273d454f1b1a6726878a74e10e7d9575bb687d4f9" exitCode=0 Nov 03 16:45:43.802413 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.802387 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" event={"ID":"f3de2da9-631c-4ff3-96d3-8cf575990923","Type":"ContainerDied","Data":"be798cf05b2c408523f3672273d454f1b1a6726878a74e10e7d9575bb687d4f9"} Nov 03 16:45:43.802651 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.802621 4947 scope.go:122] "RemoveContainer" containerID="be798cf05b2c408523f3672273d454f1b1a6726878a74e10e7d9575bb687d4f9" Nov 03 16:45:43.842749 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.815479 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb" Nov 03 16:45:43.884127 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.879410 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" event={"ID":"1cedd499-42b4-49bc-937b-e86a540b9094","Type":"ContainerStarted","Data":"6d241d49021d09719d854ffff6639d956f49751735f1e5a5b807816a9c1248f3"} Nov 03 16:45:43.993202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.992112 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3de2da9-631c-4ff3-96d3-8cf575990923-kube-api-access-jblr2\" (UniqueName: \"kubernetes.io/projected/f3de2da9-631c-4ff3-96d3-8cf575990923-kube-api-access-jblr2\") pod \"f3de2da9-631c-4ff3-96d3-8cf575990923\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " Nov 03 16:45:43.993202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.992215 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-token\" (UniqueName: \"kubernetes.io/projected/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-token\") pod \"f3de2da9-631c-4ff3-96d3-8cf575990923\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " Nov 03 16:45:43.993202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.992321 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-envoy\") pod \"f3de2da9-631c-4ff3-96d3-8cf575990923\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " Nov 03 16:45:43.993202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.992408 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-credential-socket\") pod \"f3de2da9-631c-4ff3-96d3-8cf575990923\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " Nov 03 16:45:43.993202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.992570 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-data\") pod \"f3de2da9-631c-4ff3-96d3-8cf575990923\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " Nov 03 16:45:43.993202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.992608 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3de2da9-631c-4ff3-96d3-8cf575990923-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3de2da9-631c-4ff3-96d3-8cf575990923-istiod-ca-cert\") pod \"f3de2da9-631c-4ff3-96d3-8cf575990923\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " Nov 03 16:45:43.993202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.992646 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-workload-certs\") pod \"f3de2da9-631c-4ff3-96d3-8cf575990923\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " Nov 03 16:45:43.994891 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.994548 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-credential-socket" pod "f3de2da9-631c-4ff3-96d3-8cf575990923" (UID: "f3de2da9-631c-4ff3-96d3-8cf575990923"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:43.995304 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.995057 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-data" pod "f3de2da9-631c-4ff3-96d3-8cf575990923" (UID: "f3de2da9-631c-4ff3-96d3-8cf575990923"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:43.996426 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.995996 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3de2da9-631c-4ff3-96d3-8cf575990923-istiod-ca-cert" pod "f3de2da9-631c-4ff3-96d3-8cf575990923" (UID: "f3de2da9-631c-4ff3-96d3-8cf575990923"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:43.996573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:43.996398 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-workload-certs" pod "f3de2da9-631c-4ff3-96d3-8cf575990923" (UID: "f3de2da9-631c-4ff3-96d3-8cf575990923"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:44.005639 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.003043 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-podinfo\") pod \"f3de2da9-631c-4ff3-96d3-8cf575990923\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " Nov 03 16:45:44.005639 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.003098 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-workload-socket\") pod \"f3de2da9-631c-4ff3-96d3-8cf575990923\" (UID: \"f3de2da9-631c-4ff3-96d3-8cf575990923\") " Nov 03 16:45:44.005639 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.004071 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.005639 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.004095 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3de2da9-631c-4ff3-96d3-8cf575990923-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.005639 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.004110 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.005639 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.004123 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.005639 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.005205 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-workload-socket" pod "f3de2da9-631c-4ff3-96d3-8cf575990923" (UID: "f3de2da9-631c-4ff3-96d3-8cf575990923"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:44.036782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.036720 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-podinfo" pod "f3de2da9-631c-4ff3-96d3-8cf575990923" (UID: "f3de2da9-631c-4ff3-96d3-8cf575990923"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:45:44.039503 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.039450 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-token" pod "f3de2da9-631c-4ff3-96d3-8cf575990923" (UID: "f3de2da9-631c-4ff3-96d3-8cf575990923"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:44.087932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.087881 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-envoy" pod "f3de2da9-631c-4ff3-96d3-8cf575990923" (UID: "f3de2da9-631c-4ff3-96d3-8cf575990923"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:45:44.087932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.087934 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3de2da9-631c-4ff3-96d3-8cf575990923-kube-api-access-jblr2" pod "f3de2da9-631c-4ff3-96d3-8cf575990923" (UID: "f3de2da9-631c-4ff3-96d3-8cf575990923"). InnerVolumeSpecName "kube-api-access-jblr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:44.105537 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.105502 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jblr2\" (UniqueName: \"kubernetes.io/projected/f3de2da9-631c-4ff3-96d3-8cf575990923-kube-api-access-jblr2\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.105537 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.105534 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.105537 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.105544 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.105795 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.105553 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3de2da9-631c-4ff3-96d3-8cf575990923-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.105795 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.105562 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3de2da9-631c-4ff3-96d3-8cf575990923-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.199384 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.199342 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb"] Nov 03 16:45:44.411786 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.363598 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8650a6e8-openshift-default-6ffdf697cc-4z7xb"] Nov 03 16:45:44.530738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.520089 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489"] Nov 03 16:45:44.530738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.529830 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw"] Nov 03 16:45:44.590022 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.588972 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3de2da9-631c-4ff3-96d3-8cf575990923" path="/var/lib/kubelet/pods/f3de2da9-631c-4ff3-96d3-8cf575990923/volumes" Nov 03 16:45:44.638767 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.638737 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" Nov 03 16:45:44.789721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.789193 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f1da65-4f9b-4512-b1fb-66c719d89abb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f1da65-4f9b-4512-b1fb-66c719d89abb-maas-api-tls\") pod \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\" (UID: \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\") " Nov 03 16:45:44.789721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.789291 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f1da65-4f9b-4512-b1fb-66c719d89abb-kube-api-access-cbs74\" (UniqueName: \"kubernetes.io/projected/e7f1da65-4f9b-4512-b1fb-66c719d89abb-kube-api-access-cbs74\") pod \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\" (UID: \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\") " Nov 03 16:45:44.789721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.789541 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f1da65-4f9b-4512-b1fb-66c719d89abb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f1da65-4f9b-4512-b1fb-66c719d89abb-openshift-service-ca\") pod \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\" (UID: \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\") " Nov 03 16:45:44.789721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.789586 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f1da65-4f9b-4512-b1fb-66c719d89abb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f1da65-4f9b-4512-b1fb-66c719d89abb-metrics-certs\") pod \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\" (UID: \"e7f1da65-4f9b-4512-b1fb-66c719d89abb\") " Nov 03 16:45:44.802816 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.793723 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f1da65-4f9b-4512-b1fb-66c719d89abb-openshift-service-ca" pod "e7f1da65-4f9b-4512-b1fb-66c719d89abb" (UID: "e7f1da65-4f9b-4512-b1fb-66c719d89abb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:44.854008 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.851616 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f1da65-4f9b-4512-b1fb-66c719d89abb-maas-api-tls" pod "e7f1da65-4f9b-4512-b1fb-66c719d89abb" (UID: "e7f1da65-4f9b-4512-b1fb-66c719d89abb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:45:44.854477 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.854442 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f1da65-4f9b-4512-b1fb-66c719d89abb-metrics-certs" pod "e7f1da65-4f9b-4512-b1fb-66c719d89abb" (UID: "e7f1da65-4f9b-4512-b1fb-66c719d89abb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:45:44.856668 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.856624 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f1da65-4f9b-4512-b1fb-66c719d89abb-kube-api-access-cbs74" pod "e7f1da65-4f9b-4512-b1fb-66c719d89abb" (UID: "e7f1da65-4f9b-4512-b1fb-66c719d89abb"). InnerVolumeSpecName "kube-api-access-cbs74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:44.904452 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.894997 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f1da65-4f9b-4512-b1fb-66c719d89abb-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.904452 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.895038 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbs74\" (UniqueName: \"kubernetes.io/projected/e7f1da65-4f9b-4512-b1fb-66c719d89abb-kube-api-access-cbs74\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.904452 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.895052 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f1da65-4f9b-4512-b1fb-66c719d89abb-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.904452 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.895065 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f1da65-4f9b-4512-b1fb-66c719d89abb-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:44.908931 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.908899 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" event={"ID":"1cedd499-42b4-49bc-937b-e86a540b9094","Type":"ContainerStarted","Data":"847270093b7f46ebe74d4aa04c4afc3549c9c67d697666da1765d4e73f360753"} Nov 03 16:45:44.909833 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.909808 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:45:44.911198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.911170 4947 generic.go:356] "Generic (PLEG): container finished" podID="e7f1da65-4f9b-4512-b1fb-66c719d89abb" containerID="21b8362638fd1ba2174ae72765c07f0db457264f4f6e37f5c4767ba9830e7210" exitCode=0 Nov 03 16:45:44.911373 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.911358 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" event={"ID":"e7f1da65-4f9b-4512-b1fb-66c719d89abb","Type":"ContainerDied","Data":"21b8362638fd1ba2174ae72765c07f0db457264f4f6e37f5c4767ba9830e7210"} Nov 03 16:45:44.911441 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.911389 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" event={"ID":"e7f1da65-4f9b-4512-b1fb-66c719d89abb","Type":"ContainerDied","Data":"f2a43136a4459e9d415bbb278c9a07b0c73d63cfd9d7db82aa86ebec6a5fda7e"} Nov 03 16:45:44.911441 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.911410 4947 scope.go:122] "RemoveContainer" containerID="21b8362638fd1ba2174ae72765c07f0db457264f4f6e37f5c4767ba9830e7210" Nov 03 16:45:44.911561 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.911483 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4" Nov 03 16:45:44.926619 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.926595 4947 scope.go:122] "RemoveContainer" containerID="21b8362638fd1ba2174ae72765c07f0db457264f4f6e37f5c4767ba9830e7210" Nov 03 16:45:44.927163 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:45:44.927136 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b8362638fd1ba2174ae72765c07f0db457264f4f6e37f5c4767ba9830e7210\": container with ID starting with 21b8362638fd1ba2174ae72765c07f0db457264f4f6e37f5c4767ba9830e7210 not found: ID does not exist" containerID="21b8362638fd1ba2174ae72765c07f0db457264f4f6e37f5c4767ba9830e7210" Nov 03 16:45:44.927283 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.927174 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b8362638fd1ba2174ae72765c07f0db457264f4f6e37f5c4767ba9830e7210"} err="failed to get container status \"21b8362638fd1ba2174ae72765c07f0db457264f4f6e37f5c4767ba9830e7210\": rpc error: code = NotFound desc = could not find container \"21b8362638fd1ba2174ae72765c07f0db457264f4f6e37f5c4767ba9830e7210\": container with ID starting with 21b8362638fd1ba2174ae72765c07f0db457264f4f6e37f5c4767ba9830e7210 not found: ID does not exist" Nov 03 16:45:44.956512 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.956461 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" podStartSLOduration=3.237060745 podStartE2EDuration="3.956442899s" totalImagesPullingTime="719.382154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:44.956032296 +0000 UTC m=+1888.545032290" watchObservedRunningTime="2026-11-03 16:45:44.956442899 +0000 UTC m=+1888.545442868" Nov 03 16:45:44.997382 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:44.997346 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4"] Nov 03 16:45:45.001684 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:45.001657 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-73cc79-64c46bc944-qm8q4"] Nov 03 16:45:46.648721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:46.643661 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f1da65-4f9b-4512-b1fb-66c719d89abb" path="/var/lib/kubelet/pods/e7f1da65-4f9b-4512-b1fb-66c719d89abb/volumes" Nov 03 16:45:46.931534 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:46.931430 4947 generic.go:356] "Generic (PLEG): container finished" podID="3757c5e0-0c52-4c99-a008-d6d944b33983" containerID="295adc1234281ab4b599907511e063eb871c1e040e4620f7b5d77a1f62800738" exitCode=0 Nov 03 16:45:46.931534 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:46.931505 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" event={"ID":"3757c5e0-0c52-4c99-a008-d6d944b33983","Type":"ContainerDied","Data":"295adc1234281ab4b599907511e063eb871c1e040e4620f7b5d77a1f62800738"} Nov 03 16:45:46.932192 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:46.932174 4947 scope.go:122] "RemoveContainer" containerID="295adc1234281ab4b599907511e063eb871c1e040e4620f7b5d77a1f62800738" Nov 03 16:45:46.933838 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:46.933807 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:45:46.933947 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:46.933893 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:45:46.934013 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:46.933946 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:45:47.109723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:47.105520 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:45:47.941766 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:47.941729 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" event={"ID":"3757c5e0-0c52-4c99-a008-d6d944b33983","Type":"ContainerStarted","Data":"8a3813abb3b2394d2e43c34e8ea2815c8fe440429e0d45b94beb940f220288d1"} Nov 03 16:45:48.504099 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:48.504055 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:45:48.507862 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:48.507822 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 16:45:48.508059 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:48.507888 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" podUID="3757c5e0-0c52-4c99-a008-d6d944b33983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 16:45:49.509721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:49.507145 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 16:45:49.509721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:49.507237 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" podUID="3757c5e0-0c52-4c99-a008-d6d944b33983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 16:45:49.527733 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:49.520796 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489" podUID="6cfbb6cd-2794-4071-b870-fd4bd0dc97ab" containerName="payload-processing" containerID="cri-o://46dcf2588ce614a70abbe211628e5133f22269f8035d6dfabbba41d8da3c2844" gracePeriod=25 Nov 03 16:45:49.542723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:49.536774 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw" podUID="8e1e60eb-8bf7-4d39-998c-d5e488170f41" containerName="payload-pre-processing" containerID="cri-o://cef5beb2fdeb9603a12f3d01545e950b7e5d6dec667d6d31a4cde65cc061aab3" gracePeriod=25 Nov 03 16:45:49.964759 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:49.963749 4947 generic.go:356] "Generic (PLEG): container finished" podID="8e1e60eb-8bf7-4d39-998c-d5e488170f41" containerID="cef5beb2fdeb9603a12f3d01545e950b7e5d6dec667d6d31a4cde65cc061aab3" exitCode=0 Nov 03 16:45:49.964759 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:49.963833 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw" event={"ID":"8e1e60eb-8bf7-4d39-998c-d5e488170f41","Type":"ContainerDied","Data":"cef5beb2fdeb9603a12f3d01545e950b7e5d6dec667d6d31a4cde65cc061aab3"} Nov 03 16:45:49.965410 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:49.965389 4947 generic.go:356] "Generic (PLEG): container finished" podID="6cfbb6cd-2794-4071-b870-fd4bd0dc97ab" containerID="46dcf2588ce614a70abbe211628e5133f22269f8035d6dfabbba41d8da3c2844" exitCode=0 Nov 03 16:45:49.966285 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:49.966265 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489" event={"ID":"6cfbb6cd-2794-4071-b870-fd4bd0dc97ab","Type":"ContainerDied","Data":"46dcf2588ce614a70abbe211628e5133f22269f8035d6dfabbba41d8da3c2844"} Nov 03 16:45:50.056474 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.056445 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489" Nov 03 16:45:50.059641 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.059612 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw" Nov 03 16:45:50.199741 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.188641 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cfbb6cd-2794-4071-b870-fd4bd0dc97ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cfbb6cd-2794-4071-b870-fd4bd0dc97ab-plugins-config-volume\") pod \"6cfbb6cd-2794-4071-b870-fd4bd0dc97ab\" (UID: \"6cfbb6cd-2794-4071-b870-fd4bd0dc97ab\") " Nov 03 16:45:50.199741 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.188739 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e1e60eb-8bf7-4d39-998c-d5e488170f41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1e60eb-8bf7-4d39-998c-d5e488170f41-plugins-config-volume\") pod \"8e1e60eb-8bf7-4d39-998c-d5e488170f41\" (UID: \"8e1e60eb-8bf7-4d39-998c-d5e488170f41\") " Nov 03 16:45:50.199741 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.188826 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e1e60eb-8bf7-4d39-998c-d5e488170f41-kube-api-access-xbggk\" (UniqueName: \"kubernetes.io/projected/8e1e60eb-8bf7-4d39-998c-d5e488170f41-kube-api-access-xbggk\") pod \"8e1e60eb-8bf7-4d39-998c-d5e488170f41\" (UID: \"8e1e60eb-8bf7-4d39-998c-d5e488170f41\") " Nov 03 16:45:50.199741 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.189052 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cfbb6cd-2794-4071-b870-fd4bd0dc97ab-kube-api-access-wh98t\" (UniqueName: \"kubernetes.io/projected/6cfbb6cd-2794-4071-b870-fd4bd0dc97ab-kube-api-access-wh98t\") pod \"6cfbb6cd-2794-4071-b870-fd4bd0dc97ab\" (UID: \"6cfbb6cd-2794-4071-b870-fd4bd0dc97ab\") " Nov 03 16:45:50.204734 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.202413 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cfbb6cd-2794-4071-b870-fd4bd0dc97ab-plugins-config-volume" pod "6cfbb6cd-2794-4071-b870-fd4bd0dc97ab" (UID: "6cfbb6cd-2794-4071-b870-fd4bd0dc97ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:50.204734 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.202825 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e1e60eb-8bf7-4d39-998c-d5e488170f41-plugins-config-volume" pod "8e1e60eb-8bf7-4d39-998c-d5e488170f41" (UID: "8e1e60eb-8bf7-4d39-998c-d5e488170f41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:45:50.231733 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.227553 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cfbb6cd-2794-4071-b870-fd4bd0dc97ab-kube-api-access-wh98t" pod "6cfbb6cd-2794-4071-b870-fd4bd0dc97ab" (UID: "6cfbb6cd-2794-4071-b870-fd4bd0dc97ab"). InnerVolumeSpecName "kube-api-access-wh98t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:50.252342 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.252300 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e1e60eb-8bf7-4d39-998c-d5e488170f41-kube-api-access-xbggk" pod "8e1e60eb-8bf7-4d39-998c-d5e488170f41" (UID: "8e1e60eb-8bf7-4d39-998c-d5e488170f41"). InnerVolumeSpecName "kube-api-access-xbggk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:45:50.292937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.292840 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh98t\" (UniqueName: \"kubernetes.io/projected/6cfbb6cd-2794-4071-b870-fd4bd0dc97ab-kube-api-access-wh98t\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:50.292937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.292881 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cfbb6cd-2794-4071-b870-fd4bd0dc97ab-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:50.292937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.292891 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e1e60eb-8bf7-4d39-998c-d5e488170f41-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:50.292937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.292900 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbggk\" (UniqueName: \"kubernetes.io/projected/8e1e60eb-8bf7-4d39-998c-d5e488170f41-kube-api-access-xbggk\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:45:50.522759 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.519306 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 16:45:50.522759 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.519390 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" podUID="3757c5e0-0c52-4c99-a008-d6d944b33983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 16:45:50.974360 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.974269 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489" Nov 03 16:45:50.974360 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.974285 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489" event={"ID":"6cfbb6cd-2794-4071-b870-fd4bd0dc97ab","Type":"ContainerDied","Data":"a17b3919068edbfb45a8d523dba4d74adf34c4c97520c220093c5b3f9ccc48c0"} Nov 03 16:45:50.974360 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.974356 4947 scope.go:122] "RemoveContainer" containerID="46dcf2588ce614a70abbe211628e5133f22269f8035d6dfabbba41d8da3c2844" Nov 03 16:45:50.975956 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.975924 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw" event={"ID":"8e1e60eb-8bf7-4d39-998c-d5e488170f41","Type":"ContainerDied","Data":"207d70ef3427a266dcd6447b2d9aa4dc4aa1da6d5401b227354f6fa79a24b5f2"} Nov 03 16:45:50.975956 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.975950 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw" Nov 03 16:45:50.988653 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:50.988628 4947 scope.go:122] "RemoveContainer" containerID="cef5beb2fdeb9603a12f3d01545e950b7e5d6dec667d6d31a4cde65cc061aab3" Nov 03 16:45:51.035469 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.035433 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t"] Nov 03 16:45:51.036185 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.036167 4947 cpu_manager.go:414] "containerMap: removing container" podUID="6cfbb6cd-2794-4071-b870-fd4bd0dc97ab" containerName="payload-processing" Nov 03 16:45:51.036268 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.036186 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cfbb6cd-2794-4071-b870-fd4bd0dc97ab" containerName="payload-processing" Nov 03 16:45:51.036268 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.036207 4947 cpu_manager.go:414] "containerMap: removing container" podUID="e7f1da65-4f9b-4512-b1fb-66c719d89abb" containerName="maas-api" Nov 03 16:45:51.036268 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.036212 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f1da65-4f9b-4512-b1fb-66c719d89abb" containerName="maas-api" Nov 03 16:45:51.036268 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.036222 4947 cpu_manager.go:414] "containerMap: removing container" podUID="8e1e60eb-8bf7-4d39-998c-d5e488170f41" containerName="payload-pre-processing" Nov 03 16:45:51.036268 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.036227 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e1e60eb-8bf7-4d39-998c-d5e488170f41" containerName="payload-pre-processing" Nov 03 16:45:51.036268 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.036242 4947 cpu_manager.go:414] "containerMap: removing container" podUID="f3de2da9-631c-4ff3-96d3-8cf575990923" containerName="istio-proxy" Nov 03 16:45:51.036268 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.036247 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3de2da9-631c-4ff3-96d3-8cf575990923" containerName="istio-proxy" Nov 03 16:45:51.036625 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.036357 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cfbb6cd-2794-4071-b870-fd4bd0dc97ab" containerName="payload-processing" Nov 03 16:45:51.036625 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.036369 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f1da65-4f9b-4512-b1fb-66c719d89abb" containerName="maas-api" Nov 03 16:45:51.036625 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.036377 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e1e60eb-8bf7-4d39-998c-d5e488170f41" containerName="payload-pre-processing" Nov 03 16:45:51.036625 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.036384 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3de2da9-631c-4ff3-96d3-8cf575990923" containerName="istio-proxy" Nov 03 16:45:51.042350 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.042326 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.051609 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.051580 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw"] Nov 03 16:45:51.061529 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.061404 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-654c89bcff-9twhw"] Nov 03 16:45:51.068114 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.068083 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-624649-openshift-default-dockercfg-7txfp\"" Nov 03 16:45:51.070601 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.070576 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t"] Nov 03 16:45:51.106741 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.100176 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489"] Nov 03 16:45:51.136444 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.136402 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-bf6c5b6b9-jx489"] Nov 03 16:45:51.247724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.244648 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndqr7\" (UniqueName: \"kubernetes.io/projected/95645580-4766-498e-80f5-26586a5c8b18-kube-api-access-ndqr7\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.247724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.244751 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-workload-certs\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.247724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.244795 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95645580-4766-498e-80f5-26586a5c8b18-istio-token\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.247724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.244830 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95645580-4766-498e-80f5-26586a5c8b18-istiod-ca-cert\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.247724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.244872 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-istio-envoy\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.247724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.244922 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-workload-socket\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.247724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.244995 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-istio-data\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.247724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.245090 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-credential-socket\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.247724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.245147 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95645580-4766-498e-80f5-26586a5c8b18-istio-podinfo\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.354721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.351446 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndqr7\" (UniqueName: \"kubernetes.io/projected/95645580-4766-498e-80f5-26586a5c8b18-kube-api-access-ndqr7\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.354721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.351520 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-workload-certs\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.354721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.351552 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95645580-4766-498e-80f5-26586a5c8b18-istio-token\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.354721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.351580 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95645580-4766-498e-80f5-26586a5c8b18-istiod-ca-cert\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.354721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.351616 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-istio-envoy\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.354721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.351665 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-workload-socket\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.354721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.351757 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-istio-data\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.354721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.351845 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-credential-socket\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.354721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.351905 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95645580-4766-498e-80f5-26586a5c8b18-istio-podinfo\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.354721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.354155 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-workload-socket\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.354721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.354434 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-istio-data\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.354721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.354688 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-credential-socket\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.355529 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.354977 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-workload-certs\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.388917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.385999 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95645580-4766-498e-80f5-26586a5c8b18-istiod-ca-cert\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.407820 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.391731 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-istio-envoy\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.429968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.406364 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95645580-4766-498e-80f5-26586a5c8b18-istio-podinfo\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.430104 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.409408 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95645580-4766-498e-80f5-26586a5c8b18-istio-token\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.430203 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.416519 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndqr7\" (UniqueName: \"kubernetes.io/projected/95645580-4766-498e-80f5-26586a5c8b18-kube-api-access-ndqr7\") pod \"e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.506452 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.506350 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 16:45:51.507181 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.506766 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" podUID="3757c5e0-0c52-4c99-a008-d6d944b33983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 16:45:51.680053 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.679781 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:51.953937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:51.953902 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:45:52.529366 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:52.525064 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 16:45:52.529366 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:52.525116 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" podUID="3757c5e0-0c52-4c99-a008-d6d944b33983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 16:45:52.529504 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:52.529415 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t"] Nov 03 16:45:52.530560 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:45:52.530532 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95645580_4766_498e_80f5_26586a5c8b18.slice/crio-0faccdbe58f4fdb2e810a3517c2c4093c9ba0bd99602ec4c1aecf50d04eaddca WatchSource:0}: Error finding container 0faccdbe58f4fdb2e810a3517c2c4093c9ba0bd99602ec4c1aecf50d04eaddca: Status 404 returned error can't find the container with id 0faccdbe58f4fdb2e810a3517c2c4093c9ba0bd99602ec4c1aecf50d04eaddca Nov 03 16:45:52.533298 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:52.533260 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:45:52.533403 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:52.533328 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:45:52.533403 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:52.533358 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:45:52.569106 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:52.569077 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cfbb6cd-2794-4071-b870-fd4bd0dc97ab" path="/var/lib/kubelet/pods/6cfbb6cd-2794-4071-b870-fd4bd0dc97ab/volumes" Nov 03 16:45:52.569572 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:52.569557 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e1e60eb-8bf7-4d39-998c-d5e488170f41" path="/var/lib/kubelet/pods/8e1e60eb-8bf7-4d39-998c-d5e488170f41/volumes" Nov 03 16:45:53.047717 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:53.047666 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" event={"ID":"95645580-4766-498e-80f5-26586a5c8b18","Type":"ContainerStarted","Data":"b3902df6880a5f0ca22df435066c9e4a6613218565d2281a21361b0c50fb3ccb"} Nov 03 16:45:53.048167 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:53.047765 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" event={"ID":"95645580-4766-498e-80f5-26586a5c8b18","Type":"ContainerStarted","Data":"0faccdbe58f4fdb2e810a3517c2c4093c9ba0bd99602ec4c1aecf50d04eaddca"} Nov 03 16:45:53.139799 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:53.139679 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" podStartSLOduration=3.139657121 podStartE2EDuration="3.139657121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:45:53.121194149 +0000 UTC m=+1896.710194120" watchObservedRunningTime="2026-11-03 16:45:53.139657121 +0000 UTC m=+1896.728657092" Nov 03 16:45:53.519724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:53.513772 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 16:45:53.519724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:53.513838 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" podUID="3757c5e0-0c52-4c99-a008-d6d944b33983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 16:45:53.682035 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:53.681996 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:53.687842 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:53.687818 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:53.892344 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:53.892252 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:45:54.110804 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:54.110762 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:54.127724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:54.124028 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:45:54.504728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:54.504132 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 16:45:54.504728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:54.504209 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" podUID="3757c5e0-0c52-4c99-a008-d6d944b33983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 16:45:55.529887 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:55.529856 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:45:55.530305 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:55.530276 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:45:55.531175 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:45:55.531154 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:46:00.150394 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:46:00.150352 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:46:04.740487 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:04.740452 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx"] Nov 03 16:46:04.748567 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:04.748532 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx" Nov 03 16:46:04.754577 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:04.754529 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx"] Nov 03 16:46:04.779311 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:04.779284 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2cb1bb\"" Nov 03 16:46:04.779514 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:04.779388 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2cb1bb-dockercfg-c7rvt\"" Nov 03 16:46:04.787310 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:04.787277 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf6q7\" (UniqueName: \"kubernetes.io/projected/e46654f6-5985-4318-98e0-81e5f85b6a9e-kube-api-access-bf6q7\") pod \"payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx\" (UID: \"e46654f6-5985-4318-98e0-81e5f85b6a9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx" Nov 03 16:46:04.787653 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:04.787626 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e46654f6-5985-4318-98e0-81e5f85b6a9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx\" (UID: \"e46654f6-5985-4318-98e0-81e5f85b6a9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx" Nov 03 16:46:04.931425 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:04.931369 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e46654f6-5985-4318-98e0-81e5f85b6a9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx\" (UID: \"e46654f6-5985-4318-98e0-81e5f85b6a9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx" Nov 03 16:46:04.932170 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:04.932141 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf6q7\" (UniqueName: \"kubernetes.io/projected/e46654f6-5985-4318-98e0-81e5f85b6a9e-kube-api-access-bf6q7\") pod \"payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx\" (UID: \"e46654f6-5985-4318-98e0-81e5f85b6a9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx" Nov 03 16:46:05.090154 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:05.089772 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8"] Nov 03 16:46:05.093223 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:05.090729 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e46654f6-5985-4318-98e0-81e5f85b6a9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx\" (UID: \"e46654f6-5985-4318-98e0-81e5f85b6a9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx" Nov 03 16:46:05.096875 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:05.096197 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8" Nov 03 16:46:05.121617 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:05.121549 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8"] Nov 03 16:46:05.173154 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:05.171311 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf6q7\" (UniqueName: \"kubernetes.io/projected/e46654f6-5985-4318-98e0-81e5f85b6a9e-kube-api-access-bf6q7\") pod \"payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx\" (UID: \"e46654f6-5985-4318-98e0-81e5f85b6a9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx" Nov 03 16:46:05.259743 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:05.259677 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8adc48-00d3-4ed1-9499-77673978fad7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8\" (UID: \"ea8adc48-00d3-4ed1-9499-77673978fad7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8" Nov 03 16:46:05.259949 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:05.259922 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d66g2\" (UniqueName: \"kubernetes.io/projected/ea8adc48-00d3-4ed1-9499-77673978fad7-kube-api-access-d66g2\") pod \"payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8\" (UID: \"ea8adc48-00d3-4ed1-9499-77673978fad7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8" Nov 03 16:46:05.361978 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:05.361894 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d66g2\" (UniqueName: \"kubernetes.io/projected/ea8adc48-00d3-4ed1-9499-77673978fad7-kube-api-access-d66g2\") pod \"payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8\" (UID: \"ea8adc48-00d3-4ed1-9499-77673978fad7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8" Nov 03 16:46:05.362172 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:05.362150 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8adc48-00d3-4ed1-9499-77673978fad7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8\" (UID: \"ea8adc48-00d3-4ed1-9499-77673978fad7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8" Nov 03 16:46:05.362839 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:05.362821 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8adc48-00d3-4ed1-9499-77673978fad7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8\" (UID: \"ea8adc48-00d3-4ed1-9499-77673978fad7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8" Nov 03 16:46:05.368030 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:05.368011 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx" Nov 03 16:46:05.387098 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:05.387068 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d66g2\" (UniqueName: \"kubernetes.io/projected/ea8adc48-00d3-4ed1-9499-77673978fad7-kube-api-access-d66g2\") pod \"payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8\" (UID: \"ea8adc48-00d3-4ed1-9499-77673978fad7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8" Nov 03 16:46:05.436910 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:05.435990 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8" Nov 03 16:46:06.026079 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:06.018072 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx"] Nov 03 16:46:06.332447 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:06.332301 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8"] Nov 03 16:46:06.338731 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:46:06.338616 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea8adc48_00d3_4ed1_9499_77673978fad7.slice/crio-e478d768268d3ceae18c72b73ddc75290eb63c13cb5abb5969ac266585fb1276 WatchSource:0}: Error finding container e478d768268d3ceae18c72b73ddc75290eb63c13cb5abb5969ac266585fb1276: Status 404 returned error can't find the container with id e478d768268d3ceae18c72b73ddc75290eb63c13cb5abb5969ac266585fb1276 Nov 03 16:46:06.412918 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:06.412851 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8" event={"ID":"ea8adc48-00d3-4ed1-9499-77673978fad7","Type":"ContainerStarted","Data":"e478d768268d3ceae18c72b73ddc75290eb63c13cb5abb5969ac266585fb1276"} Nov 03 16:46:06.415378 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:06.415346 4947 generic.go:356] "Generic (PLEG): container finished" podID="dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" containerID="844fe8474c8538f55ea779a4f4dac92103f6697d3fd7d374ef2034c566cc27a5" exitCode=0 Nov 03 16:46:06.415531 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:06.415429 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" event={"ID":"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3","Type":"ContainerDied","Data":"844fe8474c8538f55ea779a4f4dac92103f6697d3fd7d374ef2034c566cc27a5"} Nov 03 16:46:06.416038 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:06.416017 4947 scope.go:122] "RemoveContainer" containerID="844fe8474c8538f55ea779a4f4dac92103f6697d3fd7d374ef2034c566cc27a5" Nov 03 16:46:06.418722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:06.418672 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:46:06.418819 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:06.418778 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:46:06.418875 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:06.418824 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:46:06.418971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:06.418947 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx" event={"ID":"e46654f6-5985-4318-98e0-81e5f85b6a9e","Type":"ContainerStarted","Data":"d3cd3c1e87f2d4f131ad2737debea088d22e22f200faef3d789349f4f499b250"} Nov 03 16:46:06.419016 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:06.419000 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx" event={"ID":"e46654f6-5985-4318-98e0-81e5f85b6a9e","Type":"ContainerStarted","Data":"8dc9bbabef12c3a3f1974df90de692d27232fae4e54a42845806fa3c3a591893"} Nov 03 16:46:06.419209 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:06.419193 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx" Nov 03 16:46:06.512280 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:06.511091 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx" podStartSLOduration=2.511056726 podStartE2EDuration="2.511056726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:06.509986715 +0000 UTC m=+1910.098986685" watchObservedRunningTime="2026-11-03 16:46:06.511056726 +0000 UTC m=+1910.100056696" Nov 03 16:46:07.439986 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:07.439950 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8" event={"ID":"ea8adc48-00d3-4ed1-9499-77673978fad7","Type":"ContainerStarted","Data":"1b213e09135ca6e00cbbd1a6518704b5f18713f9a12168559a83f7db59137f35"} Nov 03 16:46:07.441733 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:07.441690 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8" Nov 03 16:46:07.444655 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:07.444633 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" event={"ID":"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3","Type":"ContainerStarted","Data":"efc5789a62905426e8349a921da47c8170ae97109aded55665e55f1fb55b2545"} Nov 03 16:46:07.489669 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:07.489610 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8" podStartSLOduration=3.489593233 podStartE2EDuration="3.489593233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:46:07.485886613 +0000 UTC m=+1911.074886586" watchObservedRunningTime="2026-11-03 16:46:07.489593233 +0000 UTC m=+1911.078593206" Nov 03 16:46:08.035105 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:08.035076 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:46:08.051754 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:08.043074 4947 patch_prober.go:29] interesting pod/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 16:46:08.052481 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:08.052445 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" podUID="dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 16:46:08.992448 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:08.992414 4947 patch_prober.go:29] interesting pod/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 16:46:08.993017 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:08.992468 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" podUID="dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 16:46:09.993640 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:09.993602 4947 patch_prober.go:29] interesting pod/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 16:46:09.994212 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:09.994175 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" podUID="dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 16:46:10.991631 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:10.991599 4947 patch_prober.go:29] interesting pod/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 16:46:10.991855 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:10.991656 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" podUID="dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 16:46:11.479971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:11.468834 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:46:11.991473 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:11.991442 4947 patch_prober.go:29] interesting pod/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 16:46:11.991649 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:11.991494 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" podUID="dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 16:46:12.991759 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:12.991693 4947 patch_prober.go:29] interesting pod/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Nov 03 16:46:12.992158 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:12.991780 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" podUID="dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Nov 03 16:46:13.531463 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:13.531417 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4"] Nov 03 16:46:14.030729 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:14.026064 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:46:14.030729 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:14.030623 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:46:17.446668 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:17.446575 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx" Nov 03 16:46:17.896875 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:17.896835 4947 patch_prober.go:29] interesting pod/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 16:46:17.897083 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:17.896907 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" podUID="70c4edfb-34cc-41d4-9187-c29e1b0a9609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:46:18.532798 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:18.532746 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4" podUID="c1dd2f69-49df-4214-bb00-9f2cfe7073df" containerName="payload-pre-processing" containerID="cri-o://c928d929de117205884d355f6a37b2dd085f118f4edf07d8953ab93efbd76333" gracePeriod=25 Nov 03 16:46:19.007025 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:19.006119 4947 generic.go:356] "Generic (PLEG): container finished" podID="70c4edfb-34cc-41d4-9187-c29e1b0a9609" containerID="3b838ee3794597597e92dd792fa17ae1a73508e706404350328423a2e077100b" exitCode=0 Nov 03 16:46:19.007025 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:19.006355 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" event={"ID":"70c4edfb-34cc-41d4-9187-c29e1b0a9609","Type":"ContainerDied","Data":"3b838ee3794597597e92dd792fa17ae1a73508e706404350328423a2e077100b"} Nov 03 16:46:19.026757 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:19.016444 4947 scope.go:122] "RemoveContainer" containerID="3b838ee3794597597e92dd792fa17ae1a73508e706404350328423a2e077100b" Nov 03 16:46:19.041683 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:19.041462 4947 generic.go:356] "Generic (PLEG): container finished" podID="c1dd2f69-49df-4214-bb00-9f2cfe7073df" containerID="c928d929de117205884d355f6a37b2dd085f118f4edf07d8953ab93efbd76333" exitCode=0 Nov 03 16:46:19.041683 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:19.041603 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4" event={"ID":"c1dd2f69-49df-4214-bb00-9f2cfe7073df","Type":"ContainerDied","Data":"c928d929de117205884d355f6a37b2dd085f118f4edf07d8953ab93efbd76333"} Nov 03 16:46:19.044092 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:19.044029 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:46:19.044362 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:19.044334 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:46:19.044608 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:19.044586 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:46:19.175032 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:19.175003 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4" Nov 03 16:46:19.230027 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:19.229992 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1dd2f69-49df-4214-bb00-9f2cfe7073df-kube-api-access-g7zb8\" (UniqueName: \"kubernetes.io/projected/c1dd2f69-49df-4214-bb00-9f2cfe7073df-kube-api-access-g7zb8\") pod \"c1dd2f69-49df-4214-bb00-9f2cfe7073df\" (UID: \"c1dd2f69-49df-4214-bb00-9f2cfe7073df\") " Nov 03 16:46:19.230405 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:19.230382 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1dd2f69-49df-4214-bb00-9f2cfe7073df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1dd2f69-49df-4214-bb00-9f2cfe7073df-plugins-config-volume\") pod \"c1dd2f69-49df-4214-bb00-9f2cfe7073df\" (UID: \"c1dd2f69-49df-4214-bb00-9f2cfe7073df\") " Nov 03 16:46:19.243947 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:19.243886 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1dd2f69-49df-4214-bb00-9f2cfe7073df-plugins-config-volume" pod "c1dd2f69-49df-4214-bb00-9f2cfe7073df" (UID: "c1dd2f69-49df-4214-bb00-9f2cfe7073df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:19.314784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:19.293801 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1dd2f69-49df-4214-bb00-9f2cfe7073df-kube-api-access-g7zb8" pod "c1dd2f69-49df-4214-bb00-9f2cfe7073df" (UID: "c1dd2f69-49df-4214-bb00-9f2cfe7073df"). InnerVolumeSpecName "kube-api-access-g7zb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:19.333776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:19.333234 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7zb8\" (UniqueName: \"kubernetes.io/projected/c1dd2f69-49df-4214-bb00-9f2cfe7073df-kube-api-access-g7zb8\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:46:19.333776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:19.333275 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1dd2f69-49df-4214-bb00-9f2cfe7073df-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:46:19.710042 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:19.709950 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8" Nov 03 16:46:20.056316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:20.056278 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4" Nov 03 16:46:20.056527 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:20.056286 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4" event={"ID":"c1dd2f69-49df-4214-bb00-9f2cfe7073df","Type":"ContainerDied","Data":"0bc50a9f97c5fa7f832e3f083aa4ada7b75d7bb4b800d2d6ff50fc70c77afbae"} Nov 03 16:46:20.056527 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:20.056429 4947 scope.go:122] "RemoveContainer" containerID="c928d929de117205884d355f6a37b2dd085f118f4edf07d8953ab93efbd76333" Nov 03 16:46:20.059795 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:20.059759 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" event={"ID":"70c4edfb-34cc-41d4-9187-c29e1b0a9609","Type":"ContainerStarted","Data":"bf7bd77b017ed1b3eafef7c0a9877d57facd9d5a16afc3eb55a72fe885a158ba"} Nov 03 16:46:20.171730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:20.171668 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4"] Nov 03 16:46:20.180404 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:20.180376 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-53bd50-74db897967-bgfw4"] Nov 03 16:46:20.566509 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:20.566478 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1dd2f69-49df-4214-bb00-9f2cfe7073df" path="/var/lib/kubelet/pods/c1dd2f69-49df-4214-bb00-9f2cfe7073df/volumes" Nov 03 16:46:20.855654 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:20.855561 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:46:20.856135 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:20.855766 4947 patch_prober.go:29] interesting pod/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 16:46:20.856135 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:20.855814 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" podUID="70c4edfb-34cc-41d4-9187-c29e1b0a9609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 16:46:21.860720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:21.859235 4947 patch_prober.go:29] interesting pod/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 16:46:21.860720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:21.859313 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" podUID="70c4edfb-34cc-41d4-9187-c29e1b0a9609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 16:46:22.865718 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:22.863935 4947 patch_prober.go:29] interesting pod/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 16:46:22.865718 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:22.863998 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" podUID="70c4edfb-34cc-41d4-9187-c29e1b0a9609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 16:46:23.856050 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:23.856016 4947 patch_prober.go:29] interesting pod/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 16:46:23.856234 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:23.856068 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" podUID="70c4edfb-34cc-41d4-9187-c29e1b0a9609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 16:46:24.856058 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:24.856019 4947 patch_prober.go:29] interesting pod/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 16:46:24.856496 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:24.856081 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" podUID="70c4edfb-34cc-41d4-9187-c29e1b0a9609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 16:46:25.856047 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:25.856012 4947 patch_prober.go:29] interesting pod/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Nov 03 16:46:25.856208 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:25.856064 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" podUID="70c4edfb-34cc-41d4-9187-c29e1b0a9609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Nov 03 16:46:26.860470 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:26.860439 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:46:26.860912 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:26.860762 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:46:26.861499 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:26.861483 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:46:28.997749 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:28.996479 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:46:30.202285 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:46:30.202231 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:46:40.783852 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:40.783820 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn"] Nov 03 16:46:40.784650 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:40.784598 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" podUID="176f518a-26d8-45e1-bfd2-a1cdeba229d0" containerName="maas-api" containerID="cri-o://603db4c3b190432e2537a65e38de0e8d44cbe38b4db332b2f8c170462cb1e921" gracePeriod=30 Nov 03 16:46:41.365090 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.365063 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" Nov 03 16:46:41.423866 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.423829 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176f518a-26d8-45e1-bfd2-a1cdeba229d0-kube-api-access-lwkgq\" (UniqueName: \"kubernetes.io/projected/176f518a-26d8-45e1-bfd2-a1cdeba229d0-kube-api-access-lwkgq\") pod \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\" (UID: \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\") " Nov 03 16:46:41.424107 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.424084 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176f518a-26d8-45e1-bfd2-a1cdeba229d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/176f518a-26d8-45e1-bfd2-a1cdeba229d0-maas-api-tls\") pod \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\" (UID: \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\") " Nov 03 16:46:41.424194 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.424149 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/176f518a-26d8-45e1-bfd2-a1cdeba229d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/176f518a-26d8-45e1-bfd2-a1cdeba229d0-openshift-service-ca\") pod \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\" (UID: \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\") " Nov 03 16:46:41.424285 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.424268 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176f518a-26d8-45e1-bfd2-a1cdeba229d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/176f518a-26d8-45e1-bfd2-a1cdeba229d0-metrics-certs\") pod \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\" (UID: \"176f518a-26d8-45e1-bfd2-a1cdeba229d0\") " Nov 03 16:46:41.424548 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.424519 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/176f518a-26d8-45e1-bfd2-a1cdeba229d0-openshift-service-ca" pod "176f518a-26d8-45e1-bfd2-a1cdeba229d0" (UID: "176f518a-26d8-45e1-bfd2-a1cdeba229d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:41.425256 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.425209 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/176f518a-26d8-45e1-bfd2-a1cdeba229d0-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:46:41.443805 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.426770 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176f518a-26d8-45e1-bfd2-a1cdeba229d0-kube-api-access-lwkgq" pod "176f518a-26d8-45e1-bfd2-a1cdeba229d0" (UID: "176f518a-26d8-45e1-bfd2-a1cdeba229d0"). InnerVolumeSpecName "kube-api-access-lwkgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:41.443805 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.427140 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176f518a-26d8-45e1-bfd2-a1cdeba229d0-metrics-certs" pod "176f518a-26d8-45e1-bfd2-a1cdeba229d0" (UID: "176f518a-26d8-45e1-bfd2-a1cdeba229d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:46:41.456522 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.456482 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176f518a-26d8-45e1-bfd2-a1cdeba229d0-maas-api-tls" pod "176f518a-26d8-45e1-bfd2-a1cdeba229d0" (UID: "176f518a-26d8-45e1-bfd2-a1cdeba229d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:46:41.526804 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.526773 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/176f518a-26d8-45e1-bfd2-a1cdeba229d0-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:46:41.526804 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.526802 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwkgq\" (UniqueName: \"kubernetes.io/projected/176f518a-26d8-45e1-bfd2-a1cdeba229d0-kube-api-access-lwkgq\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:46:41.527005 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.526817 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/176f518a-26d8-45e1-bfd2-a1cdeba229d0-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:46:41.785263 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.785229 4947 generic.go:356] "Generic (PLEG): container finished" podID="290e2d53-a6ba-4406-af23-1f2304d80983" containerID="b200513447e4cd77122b250138f46c3f77883214480ceb9f50bc6b38715cd34d" exitCode=0 Nov 03 16:46:41.785760 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.785313 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" event={"ID":"290e2d53-a6ba-4406-af23-1f2304d80983","Type":"ContainerDied","Data":"b200513447e4cd77122b250138f46c3f77883214480ceb9f50bc6b38715cd34d"} Nov 03 16:46:41.785853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.785816 4947 scope.go:122] "RemoveContainer" containerID="b200513447e4cd77122b250138f46c3f77883214480ceb9f50bc6b38715cd34d" Nov 03 16:46:41.786903 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.786880 4947 generic.go:356] "Generic (PLEG): container finished" podID="176f518a-26d8-45e1-bfd2-a1cdeba229d0" containerID="603db4c3b190432e2537a65e38de0e8d44cbe38b4db332b2f8c170462cb1e921" exitCode=0 Nov 03 16:46:41.786977 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.786923 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" Nov 03 16:46:41.786977 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.786962 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" event={"ID":"176f518a-26d8-45e1-bfd2-a1cdeba229d0","Type":"ContainerDied","Data":"603db4c3b190432e2537a65e38de0e8d44cbe38b4db332b2f8c170462cb1e921"} Nov 03 16:46:41.787069 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.787012 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn" event={"ID":"176f518a-26d8-45e1-bfd2-a1cdeba229d0","Type":"ContainerDied","Data":"82a5e9a4a66cf8cb7b81b249021a3b55672dc5a54d75ab66da720f98d2f0e44f"} Nov 03 16:46:41.787069 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.787061 4947 scope.go:122] "RemoveContainer" containerID="603db4c3b190432e2537a65e38de0e8d44cbe38b4db332b2f8c170462cb1e921" Nov 03 16:46:41.787807 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.787755 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:46:41.787937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.787842 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:46:41.787937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.787894 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:46:41.800841 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.800823 4947 scope.go:122] "RemoveContainer" containerID="603db4c3b190432e2537a65e38de0e8d44cbe38b4db332b2f8c170462cb1e921" Nov 03 16:46:41.801163 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:46:41.801144 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"603db4c3b190432e2537a65e38de0e8d44cbe38b4db332b2f8c170462cb1e921\": container with ID starting with 603db4c3b190432e2537a65e38de0e8d44cbe38b4db332b2f8c170462cb1e921 not found: ID does not exist" containerID="603db4c3b190432e2537a65e38de0e8d44cbe38b4db332b2f8c170462cb1e921" Nov 03 16:46:41.801235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.801178 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"603db4c3b190432e2537a65e38de0e8d44cbe38b4db332b2f8c170462cb1e921"} err="failed to get container status \"603db4c3b190432e2537a65e38de0e8d44cbe38b4db332b2f8c170462cb1e921\": rpc error: code = NotFound desc = could not find container \"603db4c3b190432e2537a65e38de0e8d44cbe38b4db332b2f8c170462cb1e921\": container with ID starting with 603db4c3b190432e2537a65e38de0e8d44cbe38b4db332b2f8c170462cb1e921 not found: ID does not exist" Nov 03 16:46:41.887058 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.887032 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn"] Nov 03 16:46:41.892512 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:41.892484 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2aa2ad-66b7495f87-vc9gn"] Nov 03 16:46:42.547843 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:42.547811 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:46:42.567245 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:42.567212 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176f518a-26d8-45e1-bfd2-a1cdeba229d0" path="/var/lib/kubelet/pods/176f518a-26d8-45e1-bfd2-a1cdeba229d0/volumes" Nov 03 16:46:42.795723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:42.795677 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" event={"ID":"290e2d53-a6ba-4406-af23-1f2304d80983","Type":"ContainerStarted","Data":"cf62eafeaa17334da6ef33adb2426559d9f14fe994513b99be96ff95d4a7dfdb"} Nov 03 16:46:42.983440 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:42.983344 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst"] Nov 03 16:46:42.983637 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:42.983551 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" podUID="dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" containerName="istio-proxy" containerID="cri-o://efc5789a62905426e8349a921da47c8170ae97109aded55665e55f1fb55b2545" gracePeriod=30 Nov 03 16:46:43.701029 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:43.700998 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:46:43.701384 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:43.701363 4947 patch_prober.go:29] interesting pod/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 16:46:43.702342 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:43.701527 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" podUID="290e2d53-a6ba-4406-af23-1f2304d80983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 16:46:44.033246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:44.033206 4947 patch_prober.go:29] interesting pod/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 03 16:46:44.033897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:44.033858 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" podUID="dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:46:44.713722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:44.713205 4947 patch_prober.go:29] interesting pod/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 16:46:44.713722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:44.713282 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" podUID="290e2d53-a6ba-4406-af23-1f2304d80983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 16:46:45.703167 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:45.703124 4947 patch_prober.go:29] interesting pod/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 16:46:45.703801 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:45.703761 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" podUID="290e2d53-a6ba-4406-af23-1f2304d80983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 16:46:46.751734 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:46.746070 4947 patch_prober.go:29] interesting pod/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 16:46:46.751734 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:46.746149 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" podUID="290e2d53-a6ba-4406-af23-1f2304d80983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 16:46:47.719724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:47.709076 4947 patch_prober.go:29] interesting pod/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 16:46:47.719724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:47.709136 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" podUID="290e2d53-a6ba-4406-af23-1f2304d80983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 16:46:48.678770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.677894 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:46:48.699865 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.699782 4947 patch_prober.go:29] interesting pod/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 16:46:48.699865 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.699850 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" podUID="290e2d53-a6ba-4406-af23-1f2304d80983" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 16:46:48.784143 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.784105 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istiod-ca-cert\") pod \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " Nov 03 16:46:48.784851 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.784578 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istiod-ca-cert" pod "dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" (UID: "dc5849aa-2bda-4812-bdcc-39bfb3be4ee3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:46:48.784851 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.784844 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-credential-socket\") pod \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " Nov 03 16:46:48.785059 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.784944 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-podinfo\") pod \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " Nov 03 16:46:48.785180 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.785155 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-credential-socket" pod "dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" (UID: "dc5849aa-2bda-4812-bdcc-39bfb3be4ee3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:48.785507 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.785480 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-workload-socket\") pod \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " Nov 03 16:46:48.785608 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.785525 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-token\" (UniqueName: \"kubernetes.io/projected/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-token\") pod \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " Nov 03 16:46:48.785713 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.785634 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-workload-certs\") pod \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " Nov 03 16:46:48.785713 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.785669 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-data\") pod \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " Nov 03 16:46:48.786025 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.786000 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-workload-certs" pod "dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" (UID: "dc5849aa-2bda-4812-bdcc-39bfb3be4ee3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:48.786115 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.786051 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-workload-socket" pod "dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" (UID: "dc5849aa-2bda-4812-bdcc-39bfb3be4ee3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:48.786205 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.786186 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-data" pod "dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" (UID: "dc5849aa-2bda-4812-bdcc-39bfb3be4ee3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:48.786269 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.786225 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-envoy\") pod \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " Nov 03 16:46:48.786331 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.786271 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-kube-api-access-ct8ln\" (UniqueName: \"kubernetes.io/projected/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-kube-api-access-ct8ln\") pod \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\" (UID: \"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3\") " Nov 03 16:46:48.787662 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.787638 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:46:48.787817 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.787667 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:46:48.787817 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.787681 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:46:48.787817 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.787764 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:46:48.787817 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.787780 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:46:48.795856 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.795662 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-podinfo" pod "dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" (UID: "dc5849aa-2bda-4812-bdcc-39bfb3be4ee3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:46:48.795856 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.795803 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-envoy" pod "dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" (UID: "dc5849aa-2bda-4812-bdcc-39bfb3be4ee3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:46:48.796216 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.796181 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-token" pod "dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" (UID: "dc5849aa-2bda-4812-bdcc-39bfb3be4ee3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:48.797318 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.797294 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-kube-api-access-ct8ln" pod "dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" (UID: "dc5849aa-2bda-4812-bdcc-39bfb3be4ee3"). InnerVolumeSpecName "kube-api-access-ct8ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:46:48.889673 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.889637 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:46:48.889673 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.889671 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct8ln\" (UniqueName: \"kubernetes.io/projected/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-kube-api-access-ct8ln\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:46:48.889673 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.889681 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:46:48.889949 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.889691 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:46:48.992797 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.991588 4947 generic.go:356] "Generic (PLEG): container finished" podID="dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" containerID="efc5789a62905426e8349a921da47c8170ae97109aded55665e55f1fb55b2545" exitCode=0 Nov 03 16:46:48.992797 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:48.992352 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" Nov 03 16:46:49.004966 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:49.004085 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" event={"ID":"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3","Type":"ContainerDied","Data":"efc5789a62905426e8349a921da47c8170ae97109aded55665e55f1fb55b2545"} Nov 03 16:46:49.004966 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:49.004169 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst" event={"ID":"dc5849aa-2bda-4812-bdcc-39bfb3be4ee3","Type":"ContainerDied","Data":"52f68beeb39211984060a69a3b72a26a9323a6c10052d5cca701a1b85d8a7961"} Nov 03 16:46:49.004966 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:49.004205 4947 scope.go:122] "RemoveContainer" containerID="efc5789a62905426e8349a921da47c8170ae97109aded55665e55f1fb55b2545" Nov 03 16:46:49.043093 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:49.043068 4947 scope.go:122] "RemoveContainer" containerID="844fe8474c8538f55ea779a4f4dac92103f6697d3fd7d374ef2034c566cc27a5" Nov 03 16:46:49.055133 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:49.055108 4947 scope.go:122] "RemoveContainer" containerID="efc5789a62905426e8349a921da47c8170ae97109aded55665e55f1fb55b2545" Nov 03 16:46:49.055455 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:46:49.055434 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc5789a62905426e8349a921da47c8170ae97109aded55665e55f1fb55b2545\": container with ID starting with efc5789a62905426e8349a921da47c8170ae97109aded55665e55f1fb55b2545 not found: ID does not exist" containerID="efc5789a62905426e8349a921da47c8170ae97109aded55665e55f1fb55b2545" Nov 03 16:46:49.055518 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:49.055465 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc5789a62905426e8349a921da47c8170ae97109aded55665e55f1fb55b2545"} err="failed to get container status \"efc5789a62905426e8349a921da47c8170ae97109aded55665e55f1fb55b2545\": rpc error: code = NotFound desc = could not find container \"efc5789a62905426e8349a921da47c8170ae97109aded55665e55f1fb55b2545\": container with ID starting with efc5789a62905426e8349a921da47c8170ae97109aded55665e55f1fb55b2545 not found: ID does not exist" Nov 03 16:46:49.055518 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:49.055483 4947 scope.go:122] "RemoveContainer" containerID="844fe8474c8538f55ea779a4f4dac92103f6697d3fd7d374ef2034c566cc27a5" Nov 03 16:46:49.055762 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:46:49.055744 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"844fe8474c8538f55ea779a4f4dac92103f6697d3fd7d374ef2034c566cc27a5\": container with ID starting with 844fe8474c8538f55ea779a4f4dac92103f6697d3fd7d374ef2034c566cc27a5 not found: ID does not exist" containerID="844fe8474c8538f55ea779a4f4dac92103f6697d3fd7d374ef2034c566cc27a5" Nov 03 16:46:49.055823 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:49.055767 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"844fe8474c8538f55ea779a4f4dac92103f6697d3fd7d374ef2034c566cc27a5"} err="failed to get container status \"844fe8474c8538f55ea779a4f4dac92103f6697d3fd7d374ef2034c566cc27a5\": rpc error: code = NotFound desc = could not find container \"844fe8474c8538f55ea779a4f4dac92103f6697d3fd7d374ef2034c566cc27a5\": container with ID starting with 844fe8474c8538f55ea779a4f4dac92103f6697d3fd7d374ef2034c566cc27a5 not found: ID does not exist" Nov 03 16:46:49.088970 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:49.088934 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst"] Nov 03 16:46:49.092672 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:49.092642 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2aa2ad-openshift-default-d76bbc469-h8xst"] Nov 03 16:46:49.706040 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:49.706010 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:46:49.706558 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:49.706540 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:46:49.707323 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:49.707303 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:46:50.569066 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:46:50.569036 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" path="/var/lib/kubelet/pods/dc5849aa-2bda-4812-bdcc-39bfb3be4ee3/volumes" Nov 03 16:47:00.134427 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:47:00.134388 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:47:02.249493 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:02.249450 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr"] Nov 03 16:47:02.250019 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:02.249741 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" podUID="0d73ebc6-5b5f-4d67-af5c-f424bf672754" containerName="maas-api" containerID="cri-o://afce7dcc7c22cf23a7da04792d4f7fde5653d888b123ca3a93f04442364ea081" gracePeriod=30 Nov 03 16:47:02.398360 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:02.398235 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd"] Nov 03 16:47:02.404820 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:02.400289 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd" podUID="541df157-b8b7-4031-a272-c5a80899ae76" containerName="payload-pre-processing" containerID="cri-o://90eec019b84bf8fea5e281ee4f458810bcfd3f38fccfd02c52b98be94a1bdb49" gracePeriod=30 Nov 03 16:47:02.637383 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:02.637298 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" Nov 03 16:47:02.910811 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:02.910688 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d73ebc6-5b5f-4d67-af5c-f424bf672754-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d73ebc6-5b5f-4d67-af5c-f424bf672754-openshift-service-ca\") pod \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\" (UID: \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\") " Nov 03 16:47:02.920068 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:02.920027 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d73ebc6-5b5f-4d67-af5c-f424bf672754-kube-api-access-hndnm\" (UniqueName: \"kubernetes.io/projected/0d73ebc6-5b5f-4d67-af5c-f424bf672754-kube-api-access-hndnm\") pod \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\" (UID: \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\") " Nov 03 16:47:02.920711 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:02.920659 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d73ebc6-5b5f-4d67-af5c-f424bf672754-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d73ebc6-5b5f-4d67-af5c-f424bf672754-maas-api-tls\") pod \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\" (UID: \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\") " Nov 03 16:47:02.920916 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:02.920898 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d73ebc6-5b5f-4d67-af5c-f424bf672754-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d73ebc6-5b5f-4d67-af5c-f424bf672754-metrics-certs\") pod \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\" (UID: \"0d73ebc6-5b5f-4d67-af5c-f424bf672754\") " Nov 03 16:47:02.924966 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:02.924675 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d73ebc6-5b5f-4d67-af5c-f424bf672754-kube-api-access-hndnm" pod "0d73ebc6-5b5f-4d67-af5c-f424bf672754" (UID: "0d73ebc6-5b5f-4d67-af5c-f424bf672754"). InnerVolumeSpecName "kube-api-access-hndnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:02.953199 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:02.953148 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d73ebc6-5b5f-4d67-af5c-f424bf672754-maas-api-tls" pod "0d73ebc6-5b5f-4d67-af5c-f424bf672754" (UID: "0d73ebc6-5b5f-4d67-af5c-f424bf672754"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:47:02.954248 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:02.954215 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d73ebc6-5b5f-4d67-af5c-f424bf672754-openshift-service-ca" pod "0d73ebc6-5b5f-4d67-af5c-f424bf672754" (UID: "0d73ebc6-5b5f-4d67-af5c-f424bf672754"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:02.984381 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:02.984326 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d73ebc6-5b5f-4d67-af5c-f424bf672754-metrics-certs" pod "0d73ebc6-5b5f-4d67-af5c-f424bf672754" (UID: "0d73ebc6-5b5f-4d67-af5c-f424bf672754"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:47:02.985003 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:02.984966 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd" Nov 03 16:47:03.027792 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.027735 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d73ebc6-5b5f-4d67-af5c-f424bf672754-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:03.027792 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.027769 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hndnm\" (UniqueName: \"kubernetes.io/projected/0d73ebc6-5b5f-4d67-af5c-f424bf672754-kube-api-access-hndnm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:03.027792 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.027783 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d73ebc6-5b5f-4d67-af5c-f424bf672754-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:03.027792 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.027796 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d73ebc6-5b5f-4d67-af5c-f424bf672754-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:03.128851 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.128818 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/541df157-b8b7-4031-a272-c5a80899ae76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/541df157-b8b7-4031-a272-c5a80899ae76-plugins-config-volume\") pod \"541df157-b8b7-4031-a272-c5a80899ae76\" (UID: \"541df157-b8b7-4031-a272-c5a80899ae76\") " Nov 03 16:47:03.129058 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.128871 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/541df157-b8b7-4031-a272-c5a80899ae76-kube-api-access-85g24\" (UniqueName: \"kubernetes.io/projected/541df157-b8b7-4031-a272-c5a80899ae76-kube-api-access-85g24\") pod \"541df157-b8b7-4031-a272-c5a80899ae76\" (UID: \"541df157-b8b7-4031-a272-c5a80899ae76\") " Nov 03 16:47:03.129275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.129251 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/541df157-b8b7-4031-a272-c5a80899ae76-plugins-config-volume" pod "541df157-b8b7-4031-a272-c5a80899ae76" (UID: "541df157-b8b7-4031-a272-c5a80899ae76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:03.131191 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.131165 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/541df157-b8b7-4031-a272-c5a80899ae76-kube-api-access-85g24" pod "541df157-b8b7-4031-a272-c5a80899ae76" (UID: "541df157-b8b7-4031-a272-c5a80899ae76"). InnerVolumeSpecName "kube-api-access-85g24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:03.235785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.231792 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/541df157-b8b7-4031-a272-c5a80899ae76-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:03.235785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.231841 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85g24\" (UniqueName: \"kubernetes.io/projected/541df157-b8b7-4031-a272-c5a80899ae76-kube-api-access-85g24\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:03.322420 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.322389 4947 generic.go:356] "Generic (PLEG): container finished" podID="95645580-4766-498e-80f5-26586a5c8b18" containerID="b3902df6880a5f0ca22df435066c9e4a6613218565d2281a21361b0c50fb3ccb" exitCode=0 Nov 03 16:47:03.322898 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.322462 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" event={"ID":"95645580-4766-498e-80f5-26586a5c8b18","Type":"ContainerDied","Data":"b3902df6880a5f0ca22df435066c9e4a6613218565d2281a21361b0c50fb3ccb"} Nov 03 16:47:03.322990 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.322953 4947 scope.go:122] "RemoveContainer" containerID="b3902df6880a5f0ca22df435066c9e4a6613218565d2281a21361b0c50fb3ccb" Nov 03 16:47:03.324998 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.324655 4947 generic.go:356] "Generic (PLEG): container finished" podID="541df157-b8b7-4031-a272-c5a80899ae76" containerID="90eec019b84bf8fea5e281ee4f458810bcfd3f38fccfd02c52b98be94a1bdb49" exitCode=0 Nov 03 16:47:03.324998 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.324745 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd" Nov 03 16:47:03.324998 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.324837 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd" event={"ID":"541df157-b8b7-4031-a272-c5a80899ae76","Type":"ContainerDied","Data":"90eec019b84bf8fea5e281ee4f458810bcfd3f38fccfd02c52b98be94a1bdb49"} Nov 03 16:47:03.324998 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.324875 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd" event={"ID":"541df157-b8b7-4031-a272-c5a80899ae76","Type":"ContainerDied","Data":"4ae37b994295ae78696f74b72af204127faf4ad280728688f034ea9a624e2ef3"} Nov 03 16:47:03.324998 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.324901 4947 scope.go:122] "RemoveContainer" containerID="90eec019b84bf8fea5e281ee4f458810bcfd3f38fccfd02c52b98be94a1bdb49" Nov 03 16:47:03.325229 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.325088 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:47:03.325229 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.325175 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:47:03.325229 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.325221 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:47:03.331059 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.331035 4947 generic.go:356] "Generic (PLEG): container finished" podID="0d73ebc6-5b5f-4d67-af5c-f424bf672754" containerID="afce7dcc7c22cf23a7da04792d4f7fde5653d888b123ca3a93f04442364ea081" exitCode=0 Nov 03 16:47:03.331155 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.331096 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" event={"ID":"0d73ebc6-5b5f-4d67-af5c-f424bf672754","Type":"ContainerDied","Data":"afce7dcc7c22cf23a7da04792d4f7fde5653d888b123ca3a93f04442364ea081"} Nov 03 16:47:03.331155 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.331125 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" event={"ID":"0d73ebc6-5b5f-4d67-af5c-f424bf672754","Type":"ContainerDied","Data":"bfe0465929c8004cf53847ee7aa2971cf5189e78887b3e57fb7d336944795792"} Nov 03 16:47:03.331155 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.331145 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr" Nov 03 16:47:03.345319 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.345297 4947 scope.go:122] "RemoveContainer" containerID="90eec019b84bf8fea5e281ee4f458810bcfd3f38fccfd02c52b98be94a1bdb49" Nov 03 16:47:03.345639 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:47:03.345619 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90eec019b84bf8fea5e281ee4f458810bcfd3f38fccfd02c52b98be94a1bdb49\": container with ID starting with 90eec019b84bf8fea5e281ee4f458810bcfd3f38fccfd02c52b98be94a1bdb49 not found: ID does not exist" containerID="90eec019b84bf8fea5e281ee4f458810bcfd3f38fccfd02c52b98be94a1bdb49" Nov 03 16:47:03.345764 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.345649 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90eec019b84bf8fea5e281ee4f458810bcfd3f38fccfd02c52b98be94a1bdb49"} err="failed to get container status \"90eec019b84bf8fea5e281ee4f458810bcfd3f38fccfd02c52b98be94a1bdb49\": rpc error: code = NotFound desc = could not find container \"90eec019b84bf8fea5e281ee4f458810bcfd3f38fccfd02c52b98be94a1bdb49\": container with ID starting with 90eec019b84bf8fea5e281ee4f458810bcfd3f38fccfd02c52b98be94a1bdb49 not found: ID does not exist" Nov 03 16:47:03.345764 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.345664 4947 scope.go:122] "RemoveContainer" containerID="afce7dcc7c22cf23a7da04792d4f7fde5653d888b123ca3a93f04442364ea081" Nov 03 16:47:03.355945 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.355924 4947 scope.go:122] "RemoveContainer" containerID="afce7dcc7c22cf23a7da04792d4f7fde5653d888b123ca3a93f04442364ea081" Nov 03 16:47:03.356220 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:47:03.356201 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afce7dcc7c22cf23a7da04792d4f7fde5653d888b123ca3a93f04442364ea081\": container with ID starting with afce7dcc7c22cf23a7da04792d4f7fde5653d888b123ca3a93f04442364ea081 not found: ID does not exist" containerID="afce7dcc7c22cf23a7da04792d4f7fde5653d888b123ca3a93f04442364ea081" Nov 03 16:47:03.356282 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.356227 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afce7dcc7c22cf23a7da04792d4f7fde5653d888b123ca3a93f04442364ea081"} err="failed to get container status \"afce7dcc7c22cf23a7da04792d4f7fde5653d888b123ca3a93f04442364ea081\": rpc error: code = NotFound desc = could not find container \"afce7dcc7c22cf23a7da04792d4f7fde5653d888b123ca3a93f04442364ea081\": container with ID starting with afce7dcc7c22cf23a7da04792d4f7fde5653d888b123ca3a93f04442364ea081 not found: ID does not exist" Nov 03 16:47:03.420849 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.419805 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd"] Nov 03 16:47:03.425398 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.425379 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-156c2a-865bb85768-29qfd"] Nov 03 16:47:03.460387 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.459617 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr"] Nov 03 16:47:03.464836 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:03.464808 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-156c2a-5989cc8cf8-bmbrr"] Nov 03 16:47:04.346245 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:04.346204 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" event={"ID":"95645580-4766-498e-80f5-26586a5c8b18","Type":"ContainerStarted","Data":"08f7ae79b9b8dc1f6e3c3cf5710ac6b78aa94956a701968d4f61b4b9101b9825"} Nov 03 16:47:04.565968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:04.565938 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d73ebc6-5b5f-4d67-af5c-f424bf672754" path="/var/lib/kubelet/pods/0d73ebc6-5b5f-4d67-af5c-f424bf672754/volumes" Nov 03 16:47:04.566449 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:04.566435 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="541df157-b8b7-4031-a272-c5a80899ae76" path="/var/lib/kubelet/pods/541df157-b8b7-4031-a272-c5a80899ae76/volumes" Nov 03 16:47:04.681177 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:04.681091 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:47:04.682086 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:04.682059 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 16:47:04.682197 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:04.682109 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" podUID="95645580-4766-498e-80f5-26586a5c8b18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 16:47:05.682031 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:05.681993 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 16:47:05.682439 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:05.682059 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" podUID="95645580-4766-498e-80f5-26586a5c8b18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 16:47:06.684724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:06.682069 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 16:47:06.684724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:06.682126 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" podUID="95645580-4766-498e-80f5-26586a5c8b18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 16:47:07.681539 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:07.681506 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 16:47:07.681769 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:07.681568 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" podUID="95645580-4766-498e-80f5-26586a5c8b18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 16:47:08.681401 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:08.681363 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 16:47:08.681854 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:08.681428 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" podUID="95645580-4766-498e-80f5-26586a5c8b18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 16:47:09.117723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:09.116982 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:47:09.753049 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:09.753015 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:47:09.755325 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:09.755302 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:47:15.512590 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:15.508302 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:47:24.455227 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:24.455178 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n"] Nov 03 16:47:24.456886 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:24.456856 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" podUID="1cedd499-42b4-49bc-937b-e86a540b9094" containerName="maas-api" containerID="cri-o://847270093b7f46ebe74d4aa04c4afc3549c9c67d697666da1765d4e73f360753" gracePeriod=30 Nov 03 16:47:24.758663 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:24.758627 4947 generic.go:356] "Generic (PLEG): container finished" podID="1cedd499-42b4-49bc-937b-e86a540b9094" containerID="847270093b7f46ebe74d4aa04c4afc3549c9c67d697666da1765d4e73f360753" exitCode=0 Nov 03 16:47:24.758853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:24.758836 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" event={"ID":"1cedd499-42b4-49bc-937b-e86a540b9094","Type":"ContainerDied","Data":"847270093b7f46ebe74d4aa04c4afc3549c9c67d697666da1765d4e73f360753"} Nov 03 16:47:24.798082 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:24.798054 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:47:24.915521 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:24.915479 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cedd499-42b4-49bc-937b-e86a540b9094-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cedd499-42b4-49bc-937b-e86a540b9094-maas-api-tls\") pod \"1cedd499-42b4-49bc-937b-e86a540b9094\" (UID: \"1cedd499-42b4-49bc-937b-e86a540b9094\") " Nov 03 16:47:24.915741 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:24.915643 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cedd499-42b4-49bc-937b-e86a540b9094-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cedd499-42b4-49bc-937b-e86a540b9094-openshift-service-ca\") pod \"1cedd499-42b4-49bc-937b-e86a540b9094\" (UID: \"1cedd499-42b4-49bc-937b-e86a540b9094\") " Nov 03 16:47:24.915741 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:24.915720 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cedd499-42b4-49bc-937b-e86a540b9094-kube-api-access-wtmss\" (UniqueName: \"kubernetes.io/projected/1cedd499-42b4-49bc-937b-e86a540b9094-kube-api-access-wtmss\") pod \"1cedd499-42b4-49bc-937b-e86a540b9094\" (UID: \"1cedd499-42b4-49bc-937b-e86a540b9094\") " Nov 03 16:47:24.915972 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:24.915955 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cedd499-42b4-49bc-937b-e86a540b9094-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cedd499-42b4-49bc-937b-e86a540b9094-metrics-certs\") pod \"1cedd499-42b4-49bc-937b-e86a540b9094\" (UID: \"1cedd499-42b4-49bc-937b-e86a540b9094\") " Nov 03 16:47:24.916520 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:24.916054 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cedd499-42b4-49bc-937b-e86a540b9094-openshift-service-ca" pod "1cedd499-42b4-49bc-937b-e86a540b9094" (UID: "1cedd499-42b4-49bc-937b-e86a540b9094"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:24.916634 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:24.916554 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cedd499-42b4-49bc-937b-e86a540b9094-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:24.918014 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:24.917980 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cedd499-42b4-49bc-937b-e86a540b9094-maas-api-tls" pod "1cedd499-42b4-49bc-937b-e86a540b9094" (UID: "1cedd499-42b4-49bc-937b-e86a540b9094"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:47:24.918603 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:24.918571 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cedd499-42b4-49bc-937b-e86a540b9094-metrics-certs" pod "1cedd499-42b4-49bc-937b-e86a540b9094" (UID: "1cedd499-42b4-49bc-937b-e86a540b9094"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:47:24.918603 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:24.918582 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cedd499-42b4-49bc-937b-e86a540b9094-kube-api-access-wtmss" pod "1cedd499-42b4-49bc-937b-e86a540b9094" (UID: "1cedd499-42b4-49bc-937b-e86a540b9094"). InnerVolumeSpecName "kube-api-access-wtmss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:25.107795 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:25.100017 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cedd499-42b4-49bc-937b-e86a540b9094-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:25.107795 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:25.100051 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cedd499-42b4-49bc-937b-e86a540b9094-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:25.107795 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:25.100065 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtmss\" (UniqueName: \"kubernetes.io/projected/1cedd499-42b4-49bc-937b-e86a540b9094-kube-api-access-wtmss\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:25.767048 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:25.767014 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" Nov 03 16:47:25.767048 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:25.767043 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n" event={"ID":"1cedd499-42b4-49bc-937b-e86a540b9094","Type":"ContainerDied","Data":"6d241d49021d09719d854ffff6639d956f49751735f1e5a5b807816a9c1248f3"} Nov 03 16:47:25.767539 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:25.767111 4947 scope.go:122] "RemoveContainer" containerID="847270093b7f46ebe74d4aa04c4afc3549c9c67d697666da1765d4e73f360753" Nov 03 16:47:25.824119 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:25.824088 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n"] Nov 03 16:47:25.836736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:25.836690 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-53bd50-555b6dbddb-lrp9n"] Nov 03 16:47:26.635834 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:26.635798 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cedd499-42b4-49bc-937b-e86a540b9094" path="/var/lib/kubelet/pods/1cedd499-42b4-49bc-937b-e86a540b9094/volumes" Nov 03 16:47:26.636508 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:26.636487 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j"] Nov 03 16:47:26.636924 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:26.636894 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" podUID="70c4edfb-34cc-41d4-9187-c29e1b0a9609" containerName="istio-proxy" containerID="cri-o://bf7bd77b017ed1b3eafef7c0a9877d57facd9d5a16afc3eb55a72fe885a158ba" gracePeriod=30 Nov 03 16:47:26.896810 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:26.895405 4947 patch_prober.go:29] interesting pod/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:47:26.896810 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:26.896465 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" podUID="70c4edfb-34cc-41d4-9187-c29e1b0a9609" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:47:30.132777 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:47:30.132730 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:47:31.931295 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:31.931229 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:47:31.933404 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:31.933378 4947 generic.go:356] "Generic (PLEG): container finished" podID="70c4edfb-34cc-41d4-9187-c29e1b0a9609" containerID="bf7bd77b017ed1b3eafef7c0a9877d57facd9d5a16afc3eb55a72fe885a158ba" exitCode=0 Nov 03 16:47:31.933525 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:31.933414 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" Nov 03 16:47:31.933525 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:31.933458 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" event={"ID":"70c4edfb-34cc-41d4-9187-c29e1b0a9609","Type":"ContainerDied","Data":"bf7bd77b017ed1b3eafef7c0a9877d57facd9d5a16afc3eb55a72fe885a158ba"} Nov 03 16:47:31.933525 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:31.933516 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j" event={"ID":"70c4edfb-34cc-41d4-9187-c29e1b0a9609","Type":"ContainerDied","Data":"44476f7c0332b66d64eaa9c24716c68e4eeca1d9743bc204c88b97baa4183432"} Nov 03 16:47:31.933643 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:31.933541 4947 scope.go:122] "RemoveContainer" containerID="bf7bd77b017ed1b3eafef7c0a9877d57facd9d5a16afc3eb55a72fe885a158ba" Nov 03 16:47:31.943824 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:31.943802 4947 scope.go:122] "RemoveContainer" containerID="3b838ee3794597597e92dd792fa17ae1a73508e706404350328423a2e077100b" Nov 03 16:47:31.953797 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:31.953776 4947 scope.go:122] "RemoveContainer" containerID="bf7bd77b017ed1b3eafef7c0a9877d57facd9d5a16afc3eb55a72fe885a158ba" Nov 03 16:47:31.954097 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:47:31.954079 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf7bd77b017ed1b3eafef7c0a9877d57facd9d5a16afc3eb55a72fe885a158ba\": container with ID starting with bf7bd77b017ed1b3eafef7c0a9877d57facd9d5a16afc3eb55a72fe885a158ba not found: ID does not exist" containerID="bf7bd77b017ed1b3eafef7c0a9877d57facd9d5a16afc3eb55a72fe885a158ba" Nov 03 16:47:31.954143 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:31.954107 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf7bd77b017ed1b3eafef7c0a9877d57facd9d5a16afc3eb55a72fe885a158ba"} err="failed to get container status \"bf7bd77b017ed1b3eafef7c0a9877d57facd9d5a16afc3eb55a72fe885a158ba\": rpc error: code = NotFound desc = could not find container \"bf7bd77b017ed1b3eafef7c0a9877d57facd9d5a16afc3eb55a72fe885a158ba\": container with ID starting with bf7bd77b017ed1b3eafef7c0a9877d57facd9d5a16afc3eb55a72fe885a158ba not found: ID does not exist" Nov 03 16:47:31.954143 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:31.954126 4947 scope.go:122] "RemoveContainer" containerID="3b838ee3794597597e92dd792fa17ae1a73508e706404350328423a2e077100b" Nov 03 16:47:31.954390 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:47:31.954371 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b838ee3794597597e92dd792fa17ae1a73508e706404350328423a2e077100b\": container with ID starting with 3b838ee3794597597e92dd792fa17ae1a73508e706404350328423a2e077100b not found: ID does not exist" containerID="3b838ee3794597597e92dd792fa17ae1a73508e706404350328423a2e077100b" Nov 03 16:47:31.954441 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:31.954397 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b838ee3794597597e92dd792fa17ae1a73508e706404350328423a2e077100b"} err="failed to get container status \"3b838ee3794597597e92dd792fa17ae1a73508e706404350328423a2e077100b\": rpc error: code = NotFound desc = could not find container \"3b838ee3794597597e92dd792fa17ae1a73508e706404350328423a2e077100b\": container with ID starting with 3b838ee3794597597e92dd792fa17ae1a73508e706404350328423a2e077100b not found: ID does not exist" Nov 03 16:47:32.107971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.107940 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-data\") pod \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " Nov 03 16:47:32.108159 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.107996 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-workload-socket\") pod \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " Nov 03 16:47:32.108159 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.108051 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-podinfo\") pod \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " Nov 03 16:47:32.108159 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.108076 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-token\" (UniqueName: \"kubernetes.io/projected/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-token\") pod \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " Nov 03 16:47:32.108326 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.108218 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-envoy\") pod \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " Nov 03 16:47:32.108326 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.108249 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-data" pod "70c4edfb-34cc-41d4-9187-c29e1b0a9609" (UID: "70c4edfb-34cc-41d4-9187-c29e1b0a9609"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:32.108326 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.108314 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-workload-socket" pod "70c4edfb-34cc-41d4-9187-c29e1b0a9609" (UID: "70c4edfb-34cc-41d4-9187-c29e1b0a9609"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:32.108514 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.108499 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-workload-certs\") pod \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " Nov 03 16:47:32.108634 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.108594 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istiod-ca-cert\") pod \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " Nov 03 16:47:32.108634 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.108624 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-credential-socket\") pod \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " Nov 03 16:47:32.108796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.108729 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c4edfb-34cc-41d4-9187-c29e1b0a9609-kube-api-access-sd5zn\" (UniqueName: \"kubernetes.io/projected/70c4edfb-34cc-41d4-9187-c29e1b0a9609-kube-api-access-sd5zn\") pod \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\" (UID: \"70c4edfb-34cc-41d4-9187-c29e1b0a9609\") " Nov 03 16:47:32.108850 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.108818 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-workload-certs" pod "70c4edfb-34cc-41d4-9187-c29e1b0a9609" (UID: "70c4edfb-34cc-41d4-9187-c29e1b0a9609"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:32.109252 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.109221 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-credential-socket" pod "70c4edfb-34cc-41d4-9187-c29e1b0a9609" (UID: "70c4edfb-34cc-41d4-9187-c29e1b0a9609"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:32.109252 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.109231 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istiod-ca-cert" pod "70c4edfb-34cc-41d4-9187-c29e1b0a9609" (UID: "70c4edfb-34cc-41d4-9187-c29e1b0a9609"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:32.109388 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.109300 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:32.109388 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.109320 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:32.109388 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.109334 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:32.110655 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.110632 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-token" pod "70c4edfb-34cc-41d4-9187-c29e1b0a9609" (UID: "70c4edfb-34cc-41d4-9187-c29e1b0a9609"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:32.110845 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.110653 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-envoy" pod "70c4edfb-34cc-41d4-9187-c29e1b0a9609" (UID: "70c4edfb-34cc-41d4-9187-c29e1b0a9609"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:32.110845 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.110758 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-podinfo" pod "70c4edfb-34cc-41d4-9187-c29e1b0a9609" (UID: "70c4edfb-34cc-41d4-9187-c29e1b0a9609"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:47:32.111172 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.111155 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c4edfb-34cc-41d4-9187-c29e1b0a9609-kube-api-access-sd5zn" pod "70c4edfb-34cc-41d4-9187-c29e1b0a9609" (UID: "70c4edfb-34cc-41d4-9187-c29e1b0a9609"). InnerVolumeSpecName "kube-api-access-sd5zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:32.211058 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.210962 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:32.211058 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.210995 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:32.211058 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.211008 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70c4edfb-34cc-41d4-9187-c29e1b0a9609-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:32.211058 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.211021 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd5zn\" (UniqueName: \"kubernetes.io/projected/70c4edfb-34cc-41d4-9187-c29e1b0a9609-kube-api-access-sd5zn\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:32.211058 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.211033 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:32.211058 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.211047 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70c4edfb-34cc-41d4-9187-c29e1b0a9609-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:32.286863 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.286827 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j"] Nov 03 16:47:32.289950 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.289922 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-53bd50-openshift-default-857b6dcc75-hbp9j"] Nov 03 16:47:32.565898 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:32.565870 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70c4edfb-34cc-41d4-9187-c29e1b0a9609" path="/var/lib/kubelet/pods/70c4edfb-34cc-41d4-9187-c29e1b0a9609/volumes" Nov 03 16:47:41.994861 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:41.994827 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:47:43.172722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.172087 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95"] Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174447 4947 cpu_manager.go:414] "containerMap: removing container" podUID="70c4edfb-34cc-41d4-9187-c29e1b0a9609" containerName="istio-proxy" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174478 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c4edfb-34cc-41d4-9187-c29e1b0a9609" containerName="istio-proxy" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174503 4947 cpu_manager.go:414] "containerMap: removing container" podUID="1cedd499-42b4-49bc-937b-e86a540b9094" containerName="maas-api" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174512 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cedd499-42b4-49bc-937b-e86a540b9094" containerName="maas-api" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174523 4947 cpu_manager.go:414] "containerMap: removing container" podUID="dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" containerName="istio-proxy" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174529 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" containerName="istio-proxy" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174539 4947 cpu_manager.go:414] "containerMap: removing container" podUID="70c4edfb-34cc-41d4-9187-c29e1b0a9609" containerName="istio-proxy" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174546 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c4edfb-34cc-41d4-9187-c29e1b0a9609" containerName="istio-proxy" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174569 4947 cpu_manager.go:414] "containerMap: removing container" podUID="dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" containerName="istio-proxy" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174576 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" containerName="istio-proxy" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174599 4947 cpu_manager.go:414] "containerMap: removing container" podUID="541df157-b8b7-4031-a272-c5a80899ae76" containerName="payload-pre-processing" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174608 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541df157-b8b7-4031-a272-c5a80899ae76" containerName="payload-pre-processing" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174630 4947 cpu_manager.go:414] "containerMap: removing container" podUID="176f518a-26d8-45e1-bfd2-a1cdeba229d0" containerName="maas-api" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174637 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176f518a-26d8-45e1-bfd2-a1cdeba229d0" containerName="maas-api" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174660 4947 cpu_manager.go:414] "containerMap: removing container" podUID="0d73ebc6-5b5f-4d67-af5c-f424bf672754" containerName="maas-api" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174667 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d73ebc6-5b5f-4d67-af5c-f424bf672754" containerName="maas-api" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174715 4947 cpu_manager.go:414] "containerMap: removing container" podUID="c1dd2f69-49df-4214-bb00-9f2cfe7073df" containerName="payload-pre-processing" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174723 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1dd2f69-49df-4214-bb00-9f2cfe7073df" containerName="payload-pre-processing" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174914 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" containerName="istio-proxy" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174931 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="541df157-b8b7-4031-a272-c5a80899ae76" containerName="payload-pre-processing" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174942 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="176f518a-26d8-45e1-bfd2-a1cdeba229d0" containerName="maas-api" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174955 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d73ebc6-5b5f-4d67-af5c-f424bf672754" containerName="maas-api" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174969 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1dd2f69-49df-4214-bb00-9f2cfe7073df" containerName="payload-pre-processing" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174982 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="70c4edfb-34cc-41d4-9187-c29e1b0a9609" containerName="istio-proxy" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.174992 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cedd499-42b4-49bc-937b-e86a540b9094" containerName="maas-api" Nov 03 16:47:43.200732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.175000 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="70c4edfb-34cc-41d4-9187-c29e1b0a9609" containerName="istio-proxy" Nov 03 16:47:43.275549 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.208340 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95" Nov 03 16:47:43.275549 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.271486 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95"] Nov 03 16:47:43.328724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.295810 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e0a112-dockercfg-rhct8\"" Nov 03 16:47:43.352791 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.330202 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e0a112\"" Nov 03 16:47:43.489768 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.484033 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68zq2\" (UniqueName: \"kubernetes.io/projected/86c014d5-5c96-4f07-b77e-40aed86389ce-kube-api-access-68zq2\") pod \"payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95\" (UID: \"86c014d5-5c96-4f07-b77e-40aed86389ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95" Nov 03 16:47:43.489768 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.484104 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c014d5-5c96-4f07-b77e-40aed86389ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95\" (UID: \"86c014d5-5c96-4f07-b77e-40aed86389ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95" Nov 03 16:47:43.585893 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.585850 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68zq2\" (UniqueName: \"kubernetes.io/projected/86c014d5-5c96-4f07-b77e-40aed86389ce-kube-api-access-68zq2\") pod \"payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95\" (UID: \"86c014d5-5c96-4f07-b77e-40aed86389ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95" Nov 03 16:47:43.586092 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.586020 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c014d5-5c96-4f07-b77e-40aed86389ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95\" (UID: \"86c014d5-5c96-4f07-b77e-40aed86389ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95" Nov 03 16:47:43.587187 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.587140 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c014d5-5c96-4f07-b77e-40aed86389ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95\" (UID: \"86c014d5-5c96-4f07-b77e-40aed86389ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95" Nov 03 16:47:43.612356 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.612320 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68zq2\" (UniqueName: \"kubernetes.io/projected/86c014d5-5c96-4f07-b77e-40aed86389ce-kube-api-access-68zq2\") pod \"payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95\" (UID: \"86c014d5-5c96-4f07-b77e-40aed86389ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95" Nov 03 16:47:43.835571 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:43.835535 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95" Nov 03 16:47:44.150061 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:47:44.150029 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c014d5_5c96_4f07_b77e_40aed86389ce.slice/crio-b70d3c4fe783abc1056f33eb4ca05d9027fd635f73baed82ea202c15288eaadd WatchSource:0}: Error finding container b70d3c4fe783abc1056f33eb4ca05d9027fd635f73baed82ea202c15288eaadd: Status 404 returned error can't find the container with id b70d3c4fe783abc1056f33eb4ca05d9027fd635f73baed82ea202c15288eaadd Nov 03 16:47:44.150280 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:44.150258 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95"] Nov 03 16:47:44.215367 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:44.215320 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95" event={"ID":"86c014d5-5c96-4f07-b77e-40aed86389ce","Type":"ContainerStarted","Data":"b70d3c4fe783abc1056f33eb4ca05d9027fd635f73baed82ea202c15288eaadd"} Nov 03 16:47:45.230930 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:45.230818 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95" event={"ID":"86c014d5-5c96-4f07-b77e-40aed86389ce","Type":"ContainerStarted","Data":"2b644ace6825f836166aa27f94a7bf80f01c2cb2d40c4afcb59dda9d326ce775"} Nov 03 16:47:45.231448 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:45.231101 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95" Nov 03 16:47:45.303663 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:45.298958 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95" podStartSLOduration=2.298938748 podStartE2EDuration="2.298938748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:47:45.27714352 +0000 UTC m=+2008.866143489" watchObservedRunningTime="2026-11-03 16:47:45.298938748 +0000 UTC m=+2008.887938718" Nov 03 16:47:46.037131 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.037090 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n"] Nov 03 16:47:46.039099 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.038301 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc5849aa-2bda-4812-bdcc-39bfb3be4ee3" containerName="istio-proxy" Nov 03 16:47:46.042947 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.042924 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.070004 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.069877 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n"] Nov 03 16:47:46.075761 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.075737 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ad7bf6-openshift-default-dockercfg-nsvgw\"" Nov 03 16:47:46.182814 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.174810 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-workload-socket\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.188039 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.187997 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdplm\" (UniqueName: \"kubernetes.io/projected/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-kube-api-access-sdplm\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.196730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.188362 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istiod-ca-cert\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.197180 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.197149 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-podinfo\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.210809 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.203957 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-envoy\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.210809 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.204063 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-credential-socket\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.210809 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.204128 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-data\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.210809 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.204182 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-token\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.210809 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.204235 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-workload-certs\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.305548 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.305464 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-envoy\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.306145 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.306123 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-credential-socket\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.306276 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.306261 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-data\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.306392 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.306379 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-token\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.306494 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.306482 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-workload-certs\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.306624 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.306610 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-workload-socket\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.306759 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.306744 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdplm\" (UniqueName: \"kubernetes.io/projected/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-kube-api-access-sdplm\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.306916 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.306902 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istiod-ca-cert\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.307075 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.307062 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-podinfo\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.315612 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.313175 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-credential-socket\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.317949 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.316104 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-data\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.318143 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.316304 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-workload-socket\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.318340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.316493 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-workload-certs\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.318465 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.317521 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istiod-ca-cert\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.323180 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.323155 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-envoy\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.331854 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.331825 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-podinfo\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.352764 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.341410 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-token\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.352764 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.347813 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdplm\" (UniqueName: \"kubernetes.io/projected/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-kube-api-access-sdplm\") pod \"e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.361280 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.361198 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:46.874349 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:47:46.874310 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe8beeb_2b56_4b2d_8d21_51c2ce71f159.slice/crio-a154e24428a064660f1bfe23819c62bab2bc9bd3cca29928b7567f8cac08767b WatchSource:0}: Error finding container a154e24428a064660f1bfe23819c62bab2bc9bd3cca29928b7567f8cac08767b: Status 404 returned error can't find the container with id a154e24428a064660f1bfe23819c62bab2bc9bd3cca29928b7567f8cac08767b Nov 03 16:47:46.875435 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.875415 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n"] Nov 03 16:47:46.877350 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.877315 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:47:46.877431 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.877384 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:47:46.877431 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:46.877411 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:47:47.310582 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:47.310544 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" event={"ID":"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159","Type":"ContainerStarted","Data":"eca4fd1e96600bd71ebe9180b7e077788b19a865d43dfb3ad3e86d9d2813f414"} Nov 03 16:47:47.311035 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:47.310635 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" event={"ID":"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159","Type":"ContainerStarted","Data":"a154e24428a064660f1bfe23819c62bab2bc9bd3cca29928b7567f8cac08767b"} Nov 03 16:47:47.360473 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:47.360419 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" podStartSLOduration=2.360405 podStartE2EDuration="2.360405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:47:47.359749222 +0000 UTC m=+2010.948749192" watchObservedRunningTime="2026-11-03 16:47:47.360405 +0000 UTC m=+2010.949404969" Nov 03 16:47:47.362181 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:47.362156 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:47.367984 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:47.367959 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:48.317604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:48.317576 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:48.318662 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:48.318644 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:47:52.448176 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:52.448137 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn"] Nov 03 16:47:52.449049 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:52.448392 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" podUID="290e2d53-a6ba-4406-af23-1f2304d80983" containerName="istio-proxy" containerID="cri-o://cf62eafeaa17334da6ef33adb2426559d9f14fe994513b99be96ff95d4a7dfdb" gracePeriod=30 Nov 03 16:47:56.302782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:56.302752 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95" Nov 03 16:47:57.873506 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.871938 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:47:57.936508 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.936475 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-istio-envoy\") pod \"290e2d53-a6ba-4406-af23-1f2304d80983\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " Nov 03 16:47:57.936685 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.936544 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-credential-socket\") pod \"290e2d53-a6ba-4406-af23-1f2304d80983\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " Nov 03 16:47:57.936829 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.936743 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/290e2d53-a6ba-4406-af23-1f2304d80983-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/290e2d53-a6ba-4406-af23-1f2304d80983-istio-podinfo\") pod \"290e2d53-a6ba-4406-af23-1f2304d80983\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " Nov 03 16:47:57.936829 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.936792 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-workload-socket\") pod \"290e2d53-a6ba-4406-af23-1f2304d80983\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " Nov 03 16:47:57.936922 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.936847 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290e2d53-a6ba-4406-af23-1f2304d80983-istio-token\" (UniqueName: \"kubernetes.io/projected/290e2d53-a6ba-4406-af23-1f2304d80983-istio-token\") pod \"290e2d53-a6ba-4406-af23-1f2304d80983\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " Nov 03 16:47:57.936922 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.936888 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290e2d53-a6ba-4406-af23-1f2304d80983-kube-api-access-t4q5g\" (UniqueName: \"kubernetes.io/projected/290e2d53-a6ba-4406-af23-1f2304d80983-kube-api-access-t4q5g\") pod \"290e2d53-a6ba-4406-af23-1f2304d80983\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " Nov 03 16:47:57.937027 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.936940 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-istio-data\") pod \"290e2d53-a6ba-4406-af23-1f2304d80983\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " Nov 03 16:47:57.937027 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.936964 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-credential-socket" pod "290e2d53-a6ba-4406-af23-1f2304d80983" (UID: "290e2d53-a6ba-4406-af23-1f2304d80983"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:57.937027 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.937002 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-workload-certs\") pod \"290e2d53-a6ba-4406-af23-1f2304d80983\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " Nov 03 16:47:57.937189 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.937038 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/290e2d53-a6ba-4406-af23-1f2304d80983-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/290e2d53-a6ba-4406-af23-1f2304d80983-istiod-ca-cert\") pod \"290e2d53-a6ba-4406-af23-1f2304d80983\" (UID: \"290e2d53-a6ba-4406-af23-1f2304d80983\") " Nov 03 16:47:57.937189 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.937041 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-workload-socket" pod "290e2d53-a6ba-4406-af23-1f2304d80983" (UID: "290e2d53-a6ba-4406-af23-1f2304d80983"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:57.937303 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.937205 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-istio-data" pod "290e2d53-a6ba-4406-af23-1f2304d80983" (UID: "290e2d53-a6ba-4406-af23-1f2304d80983"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:57.937365 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.937308 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-workload-certs" pod "290e2d53-a6ba-4406-af23-1f2304d80983" (UID: "290e2d53-a6ba-4406-af23-1f2304d80983"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:57.937519 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.937494 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290e2d53-a6ba-4406-af23-1f2304d80983-istiod-ca-cert" pod "290e2d53-a6ba-4406-af23-1f2304d80983" (UID: "290e2d53-a6ba-4406-af23-1f2304d80983"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:47:57.938089 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.938064 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:57.938202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.938091 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:57.938202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.938106 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:57.938202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.938118 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/290e2d53-a6ba-4406-af23-1f2304d80983-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:57.938202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.938131 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:57.939128 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.939105 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-istio-envoy" pod "290e2d53-a6ba-4406-af23-1f2304d80983" (UID: "290e2d53-a6ba-4406-af23-1f2304d80983"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:47:57.939491 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.939471 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290e2d53-a6ba-4406-af23-1f2304d80983-kube-api-access-t4q5g" pod "290e2d53-a6ba-4406-af23-1f2304d80983" (UID: "290e2d53-a6ba-4406-af23-1f2304d80983"). InnerVolumeSpecName "kube-api-access-t4q5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:57.939744 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.939682 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290e2d53-a6ba-4406-af23-1f2304d80983-istio-token" pod "290e2d53-a6ba-4406-af23-1f2304d80983" (UID: "290e2d53-a6ba-4406-af23-1f2304d80983"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:47:57.939744 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:57.939682 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/290e2d53-a6ba-4406-af23-1f2304d80983-istio-podinfo" pod "290e2d53-a6ba-4406-af23-1f2304d80983" (UID: "290e2d53-a6ba-4406-af23-1f2304d80983"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:47:58.039550 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.039509 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/290e2d53-a6ba-4406-af23-1f2304d80983-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:58.039550 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.039548 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/290e2d53-a6ba-4406-af23-1f2304d80983-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:58.039550 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.039560 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4q5g\" (UniqueName: \"kubernetes.io/projected/290e2d53-a6ba-4406-af23-1f2304d80983-kube-api-access-t4q5g\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:58.039864 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.039570 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/290e2d53-a6ba-4406-af23-1f2304d80983-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:47:58.318102 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.313398 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96"] Nov 03 16:47:58.318102 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.314987 4947 cpu_manager.go:414] "containerMap: removing container" podUID="290e2d53-a6ba-4406-af23-1f2304d80983" containerName="istio-proxy" Nov 03 16:47:58.318102 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.315010 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290e2d53-a6ba-4406-af23-1f2304d80983" containerName="istio-proxy" Nov 03 16:47:58.318102 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.315377 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="290e2d53-a6ba-4406-af23-1f2304d80983" containerName="istio-proxy" Nov 03 16:47:58.321639 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.321609 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:47:58.342841 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.342806 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d71e12-serving-cert\"" Nov 03 16:47:58.343014 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.342975 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l45rf\" (UniqueName: \"kubernetes.io/projected/bde4c160-1141-4f29-b691-a13cfa66d1f4-kube-api-access-l45rf\") pod \"maas-api-e2e-ipp-b-d71e12-cf75846df-87k96\" (UID: \"bde4c160-1141-4f29-b691-a13cfa66d1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:47:58.343106 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.343088 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde4c160-1141-4f29-b691-a13cfa66d1f4-metrics-certs\") pod \"maas-api-e2e-ipp-b-d71e12-cf75846df-87k96\" (UID: \"bde4c160-1141-4f29-b691-a13cfa66d1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:47:58.343165 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.343146 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde4c160-1141-4f29-b691-a13cfa66d1f4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d71e12-cf75846df-87k96\" (UID: \"bde4c160-1141-4f29-b691-a13cfa66d1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:47:58.343306 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.343293 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde4c160-1141-4f29-b691-a13cfa66d1f4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d71e12-cf75846df-87k96\" (UID: \"bde4c160-1141-4f29-b691-a13cfa66d1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:47:58.344389 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.344364 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96"] Nov 03 16:47:58.445020 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.444987 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde4c160-1141-4f29-b691-a13cfa66d1f4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d71e12-cf75846df-87k96\" (UID: \"bde4c160-1141-4f29-b691-a13cfa66d1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:47:58.445217 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.445091 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l45rf\" (UniqueName: \"kubernetes.io/projected/bde4c160-1141-4f29-b691-a13cfa66d1f4-kube-api-access-l45rf\") pod \"maas-api-e2e-ipp-b-d71e12-cf75846df-87k96\" (UID: \"bde4c160-1141-4f29-b691-a13cfa66d1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:47:58.445217 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.445179 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde4c160-1141-4f29-b691-a13cfa66d1f4-metrics-certs\") pod \"maas-api-e2e-ipp-b-d71e12-cf75846df-87k96\" (UID: \"bde4c160-1141-4f29-b691-a13cfa66d1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:47:58.445348 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.445255 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde4c160-1141-4f29-b691-a13cfa66d1f4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d71e12-cf75846df-87k96\" (UID: \"bde4c160-1141-4f29-b691-a13cfa66d1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:47:58.445414 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:47:58.445396 4947 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-serving-cert: secret "maas-api-e2e-ipp-b-d71e12-serving-cert" not found Nov 03 16:47:58.445512 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:47:58.445497 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bde4c160-1141-4f29-b691-a13cfa66d1f4-maas-api-tls podName:bde4c160-1141-4f29-b691-a13cfa66d1f4 nodeName:}" failed. No retries permitted until 2026-11-03 16:47:58.945475864 +0000 UTC m=+2022.534475827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bde4c160-1141-4f29-b691-a13cfa66d1f4-maas-api-tls") pod "maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" (UID: "bde4c160-1141-4f29-b691-a13cfa66d1f4") : secret "maas-api-e2e-ipp-b-d71e12-serving-cert" not found Nov 03 16:47:58.445802 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.445781 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde4c160-1141-4f29-b691-a13cfa66d1f4-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d71e12-cf75846df-87k96\" (UID: \"bde4c160-1141-4f29-b691-a13cfa66d1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:47:58.447822 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.447805 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde4c160-1141-4f29-b691-a13cfa66d1f4-metrics-certs\") pod \"maas-api-e2e-ipp-b-d71e12-cf75846df-87k96\" (UID: \"bde4c160-1141-4f29-b691-a13cfa66d1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:47:58.470439 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.470409 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l45rf\" (UniqueName: \"kubernetes.io/projected/bde4c160-1141-4f29-b691-a13cfa66d1f4-kube-api-access-l45rf\") pod \"maas-api-e2e-ipp-b-d71e12-cf75846df-87k96\" (UID: \"bde4c160-1141-4f29-b691-a13cfa66d1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:47:58.472783 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.472758 4947 generic.go:356] "Generic (PLEG): container finished" podID="290e2d53-a6ba-4406-af23-1f2304d80983" containerID="cf62eafeaa17334da6ef33adb2426559d9f14fe994513b99be96ff95d4a7dfdb" exitCode=0 Nov 03 16:47:58.472883 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.472802 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" event={"ID":"290e2d53-a6ba-4406-af23-1f2304d80983","Type":"ContainerDied","Data":"cf62eafeaa17334da6ef33adb2426559d9f14fe994513b99be96ff95d4a7dfdb"} Nov 03 16:47:58.472883 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.472824 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" Nov 03 16:47:58.472883 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.472845 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn" event={"ID":"290e2d53-a6ba-4406-af23-1f2304d80983","Type":"ContainerDied","Data":"b17b6366d1162731e4dd9835b87ffb4edb4d10847b31a56cdec240596fe1a58c"} Nov 03 16:47:58.472883 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.472866 4947 scope.go:122] "RemoveContainer" containerID="cf62eafeaa17334da6ef33adb2426559d9f14fe994513b99be96ff95d4a7dfdb" Nov 03 16:47:58.483730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.483687 4947 scope.go:122] "RemoveContainer" containerID="b200513447e4cd77122b250138f46c3f77883214480ceb9f50bc6b38715cd34d" Nov 03 16:47:58.493055 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.493024 4947 scope.go:122] "RemoveContainer" containerID="cf62eafeaa17334da6ef33adb2426559d9f14fe994513b99be96ff95d4a7dfdb" Nov 03 16:47:58.493365 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:47:58.493344 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf62eafeaa17334da6ef33adb2426559d9f14fe994513b99be96ff95d4a7dfdb\": container with ID starting with cf62eafeaa17334da6ef33adb2426559d9f14fe994513b99be96ff95d4a7dfdb not found: ID does not exist" containerID="cf62eafeaa17334da6ef33adb2426559d9f14fe994513b99be96ff95d4a7dfdb" Nov 03 16:47:58.493454 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.493370 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf62eafeaa17334da6ef33adb2426559d9f14fe994513b99be96ff95d4a7dfdb"} err="failed to get container status \"cf62eafeaa17334da6ef33adb2426559d9f14fe994513b99be96ff95d4a7dfdb\": rpc error: code = NotFound desc = could not find container \"cf62eafeaa17334da6ef33adb2426559d9f14fe994513b99be96ff95d4a7dfdb\": container with ID starting with cf62eafeaa17334da6ef33adb2426559d9f14fe994513b99be96ff95d4a7dfdb not found: ID does not exist" Nov 03 16:47:58.493454 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.493387 4947 scope.go:122] "RemoveContainer" containerID="b200513447e4cd77122b250138f46c3f77883214480ceb9f50bc6b38715cd34d" Nov 03 16:47:58.493663 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:47:58.493649 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b200513447e4cd77122b250138f46c3f77883214480ceb9f50bc6b38715cd34d\": container with ID starting with b200513447e4cd77122b250138f46c3f77883214480ceb9f50bc6b38715cd34d not found: ID does not exist" containerID="b200513447e4cd77122b250138f46c3f77883214480ceb9f50bc6b38715cd34d" Nov 03 16:47:58.493753 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.493666 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b200513447e4cd77122b250138f46c3f77883214480ceb9f50bc6b38715cd34d"} err="failed to get container status \"b200513447e4cd77122b250138f46c3f77883214480ceb9f50bc6b38715cd34d\": rpc error: code = NotFound desc = could not find container \"b200513447e4cd77122b250138f46c3f77883214480ceb9f50bc6b38715cd34d\": container with ID starting with b200513447e4cd77122b250138f46c3f77883214480ceb9f50bc6b38715cd34d not found: ID does not exist" Nov 03 16:47:58.534856 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.534830 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn"] Nov 03 16:47:58.538622 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.538598 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-98e3b9-openshift-default-7655c6d769-q2nbn"] Nov 03 16:47:58.566774 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.566750 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="290e2d53-a6ba-4406-af23-1f2304d80983" path="/var/lib/kubelet/pods/290e2d53-a6ba-4406-af23-1f2304d80983/volumes" Nov 03 16:47:58.962560 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.962522 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde4c160-1141-4f29-b691-a13cfa66d1f4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d71e12-cf75846df-87k96\" (UID: \"bde4c160-1141-4f29-b691-a13cfa66d1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:47:58.969282 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:58.969249 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde4c160-1141-4f29-b691-a13cfa66d1f4-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d71e12-cf75846df-87k96\" (UID: \"bde4c160-1141-4f29-b691-a13cfa66d1f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:47:59.244431 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:59.239970 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:47:59.632811 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:47:59.629434 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96"] Nov 03 16:48:00.166460 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:48:00.166420 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:48:00.491532 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:00.491504 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" event={"ID":"bde4c160-1141-4f29-b691-a13cfa66d1f4","Type":"ContainerStarted","Data":"be567175c6851e161ec6220d7a093efd59bab67ce1bd0d291e5f260dd769e701"} Nov 03 16:48:00.491723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:00.491585 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" event={"ID":"bde4c160-1141-4f29-b691-a13cfa66d1f4","Type":"ContainerStarted","Data":"cc5cfc5c9351d45b2680df7b6aa7875316aab1bef51d8aaa2775b80fcb9bf93e"} Nov 03 16:48:00.491723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:00.491640 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:48:00.538864 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:00.538800 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" podStartSLOduration=2.1424727470000002 podStartE2EDuration="2.538785652s" totalImagesPullingTime="396.312905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:00.538466583 +0000 UTC m=+2024.127466553" watchObservedRunningTime="2026-11-03 16:48:00.538785652 +0000 UTC m=+2024.127785621" Nov 03 16:48:06.504441 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:06.504411 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:48:13.610291 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:13.610253 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj"] Nov 03 16:48:13.610824 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:13.610493 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" podUID="e08739cb-9aea-48c7-ab97-6d57c629880c" containerName="maas-api" containerID="cri-o://a1713c5ed697f35e9e06d762ec9c5cd84194630dc31de2bfc0ead4a917c5981f" gracePeriod=30 Nov 03 16:48:13.761542 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:13.761507 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8"] Nov 03 16:48:13.761787 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:13.761759 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8" podUID="ea8adc48-00d3-4ed1-9499-77673978fad7" containerName="payload-processing" containerID="cri-o://1b213e09135ca6e00cbbd1a6518704b5f18713f9a12168559a83f7db59137f35" gracePeriod=30 Nov 03 16:48:13.788951 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:13.788922 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx"] Nov 03 16:48:13.789183 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:13.789154 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx" podUID="e46654f6-5985-4318-98e0-81e5f85b6a9e" containerName="payload-pre-processing" containerID="cri-o://d3cd3c1e87f2d4f131ad2737debea088d22e22f200faef3d789349f4f499b250" gracePeriod=30 Nov 03 16:48:13.968350 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:13.968317 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" Nov 03 16:48:14.048041 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.042097 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e08739cb-9aea-48c7-ab97-6d57c629880c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08739cb-9aea-48c7-ab97-6d57c629880c-openshift-service-ca\") pod \"e08739cb-9aea-48c7-ab97-6d57c629880c\" (UID: \"e08739cb-9aea-48c7-ab97-6d57c629880c\") " Nov 03 16:48:14.048041 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.042174 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e08739cb-9aea-48c7-ab97-6d57c629880c-kube-api-access-6xwnr\" (UniqueName: \"kubernetes.io/projected/e08739cb-9aea-48c7-ab97-6d57c629880c-kube-api-access-6xwnr\") pod \"e08739cb-9aea-48c7-ab97-6d57c629880c\" (UID: \"e08739cb-9aea-48c7-ab97-6d57c629880c\") " Nov 03 16:48:14.048041 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.042340 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e08739cb-9aea-48c7-ab97-6d57c629880c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e08739cb-9aea-48c7-ab97-6d57c629880c-maas-api-tls\") pod \"e08739cb-9aea-48c7-ab97-6d57c629880c\" (UID: \"e08739cb-9aea-48c7-ab97-6d57c629880c\") " Nov 03 16:48:14.048041 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.042459 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e08739cb-9aea-48c7-ab97-6d57c629880c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08739cb-9aea-48c7-ab97-6d57c629880c-metrics-certs\") pod \"e08739cb-9aea-48c7-ab97-6d57c629880c\" (UID: \"e08739cb-9aea-48c7-ab97-6d57c629880c\") " Nov 03 16:48:14.048041 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.045532 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08739cb-9aea-48c7-ab97-6d57c629880c-openshift-service-ca" pod "e08739cb-9aea-48c7-ab97-6d57c629880c" (UID: "e08739cb-9aea-48c7-ab97-6d57c629880c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:14.048041 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.047793 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e08739cb-9aea-48c7-ab97-6d57c629880c-maas-api-tls" pod "e08739cb-9aea-48c7-ab97-6d57c629880c" (UID: "e08739cb-9aea-48c7-ab97-6d57c629880c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:48:14.048657 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.048611 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e08739cb-9aea-48c7-ab97-6d57c629880c-metrics-certs" pod "e08739cb-9aea-48c7-ab97-6d57c629880c" (UID: "e08739cb-9aea-48c7-ab97-6d57c629880c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:48:14.050143 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.049453 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e08739cb-9aea-48c7-ab97-6d57c629880c-kube-api-access-6xwnr" pod "e08739cb-9aea-48c7-ab97-6d57c629880c" (UID: "e08739cb-9aea-48c7-ab97-6d57c629880c"). InnerVolumeSpecName "kube-api-access-6xwnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:14.102758 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.089406 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8" Nov 03 16:48:14.259783 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.146080 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08739cb-9aea-48c7-ab97-6d57c629880c-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:14.260039 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.260013 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xwnr\" (UniqueName: \"kubernetes.io/projected/e08739cb-9aea-48c7-ab97-6d57c629880c-kube-api-access-6xwnr\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:14.260128 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.260117 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e08739cb-9aea-48c7-ab97-6d57c629880c-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:14.260196 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.260188 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e08739cb-9aea-48c7-ab97-6d57c629880c-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:14.260267 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.158580 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx" Nov 03 16:48:14.362479 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.361258 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea8adc48-00d3-4ed1-9499-77673978fad7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8adc48-00d3-4ed1-9499-77673978fad7-plugins-config-volume\") pod \"ea8adc48-00d3-4ed1-9499-77673978fad7\" (UID: \"ea8adc48-00d3-4ed1-9499-77673978fad7\") " Nov 03 16:48:14.362479 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.361326 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e46654f6-5985-4318-98e0-81e5f85b6a9e-kube-api-access-bf6q7\" (UniqueName: \"kubernetes.io/projected/e46654f6-5985-4318-98e0-81e5f85b6a9e-kube-api-access-bf6q7\") pod \"e46654f6-5985-4318-98e0-81e5f85b6a9e\" (UID: \"e46654f6-5985-4318-98e0-81e5f85b6a9e\") " Nov 03 16:48:14.362479 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.361546 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e46654f6-5985-4318-98e0-81e5f85b6a9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e46654f6-5985-4318-98e0-81e5f85b6a9e-plugins-config-volume\") pod \"e46654f6-5985-4318-98e0-81e5f85b6a9e\" (UID: \"e46654f6-5985-4318-98e0-81e5f85b6a9e\") " Nov 03 16:48:14.362479 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.361618 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea8adc48-00d3-4ed1-9499-77673978fad7-kube-api-access-d66g2\" (UniqueName: \"kubernetes.io/projected/ea8adc48-00d3-4ed1-9499-77673978fad7-kube-api-access-d66g2\") pod \"ea8adc48-00d3-4ed1-9499-77673978fad7\" (UID: \"ea8adc48-00d3-4ed1-9499-77673978fad7\") " Nov 03 16:48:14.365502 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.365474 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e46654f6-5985-4318-98e0-81e5f85b6a9e-plugins-config-volume" pod "e46654f6-5985-4318-98e0-81e5f85b6a9e" (UID: "e46654f6-5985-4318-98e0-81e5f85b6a9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:14.365736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.365451 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea8adc48-00d3-4ed1-9499-77673978fad7-plugins-config-volume" pod "ea8adc48-00d3-4ed1-9499-77673978fad7" (UID: "ea8adc48-00d3-4ed1-9499-77673978fad7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:14.371865 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.371832 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea8adc48-00d3-4ed1-9499-77673978fad7-kube-api-access-d66g2" pod "ea8adc48-00d3-4ed1-9499-77673978fad7" (UID: "ea8adc48-00d3-4ed1-9499-77673978fad7"). InnerVolumeSpecName "kube-api-access-d66g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:14.372020 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.371996 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46654f6-5985-4318-98e0-81e5f85b6a9e-kube-api-access-bf6q7" pod "e46654f6-5985-4318-98e0-81e5f85b6a9e" (UID: "e46654f6-5985-4318-98e0-81e5f85b6a9e"). InnerVolumeSpecName "kube-api-access-bf6q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:14.463850 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.463817 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea8adc48-00d3-4ed1-9499-77673978fad7-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:14.463850 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.463848 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bf6q7\" (UniqueName: \"kubernetes.io/projected/e46654f6-5985-4318-98e0-81e5f85b6a9e-kube-api-access-bf6q7\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:14.464122 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.463857 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e46654f6-5985-4318-98e0-81e5f85b6a9e-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:14.464122 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.463876 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d66g2\" (UniqueName: \"kubernetes.io/projected/ea8adc48-00d3-4ed1-9499-77673978fad7-kube-api-access-d66g2\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:14.483704 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.483659 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq"] Nov 03 16:48:14.484689 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.484669 4947 cpu_manager.go:414] "containerMap: removing container" podUID="e46654f6-5985-4318-98e0-81e5f85b6a9e" containerName="payload-pre-processing" Nov 03 16:48:14.484771 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.484713 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e46654f6-5985-4318-98e0-81e5f85b6a9e" containerName="payload-pre-processing" Nov 03 16:48:14.484771 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.484752 4947 cpu_manager.go:414] "containerMap: removing container" podUID="ea8adc48-00d3-4ed1-9499-77673978fad7" containerName="payload-processing" Nov 03 16:48:14.484771 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.484760 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea8adc48-00d3-4ed1-9499-77673978fad7" containerName="payload-processing" Nov 03 16:48:14.484872 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.484776 4947 cpu_manager.go:414] "containerMap: removing container" podUID="290e2d53-a6ba-4406-af23-1f2304d80983" containerName="istio-proxy" Nov 03 16:48:14.484872 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.484784 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290e2d53-a6ba-4406-af23-1f2304d80983" containerName="istio-proxy" Nov 03 16:48:14.484872 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.484826 4947 cpu_manager.go:414] "containerMap: removing container" podUID="e08739cb-9aea-48c7-ab97-6d57c629880c" containerName="maas-api" Nov 03 16:48:14.484872 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.484833 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e08739cb-9aea-48c7-ab97-6d57c629880c" containerName="maas-api" Nov 03 16:48:14.485012 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.484999 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e46654f6-5985-4318-98e0-81e5f85b6a9e" containerName="payload-pre-processing" Nov 03 16:48:14.485046 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.485022 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea8adc48-00d3-4ed1-9499-77673978fad7" containerName="payload-processing" Nov 03 16:48:14.485046 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.485040 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e08739cb-9aea-48c7-ab97-6d57c629880c" containerName="maas-api" Nov 03 16:48:14.485110 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.485051 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="290e2d53-a6ba-4406-af23-1f2304d80983" containerName="istio-proxy" Nov 03 16:48:14.489409 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.489390 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" Nov 03 16:48:14.506329 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.505571 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ad7bf6-serving-cert\"" Nov 03 16:48:14.510900 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.510834 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq"] Nov 03 16:48:14.564710 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.564661 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgnrf\" (UniqueName: \"kubernetes.io/projected/f6c68b79-535a-403e-aab9-5a847fd988e8-kube-api-access-mgnrf\") pod \"maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq\" (UID: \"f6c68b79-535a-403e-aab9-5a847fd988e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" Nov 03 16:48:14.564920 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.564721 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6c68b79-535a-403e-aab9-5a847fd988e8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq\" (UID: \"f6c68b79-535a-403e-aab9-5a847fd988e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" Nov 03 16:48:14.564920 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.564855 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6c68b79-535a-403e-aab9-5a847fd988e8-metrics-certs\") pod \"maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq\" (UID: \"f6c68b79-535a-403e-aab9-5a847fd988e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" Nov 03 16:48:14.565122 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.565103 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c68b79-535a-403e-aab9-5a847fd988e8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq\" (UID: \"f6c68b79-535a-403e-aab9-5a847fd988e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" Nov 03 16:48:14.623225 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.623192 4947 generic.go:356] "Generic (PLEG): container finished" podID="e46654f6-5985-4318-98e0-81e5f85b6a9e" containerID="d3cd3c1e87f2d4f131ad2737debea088d22e22f200faef3d789349f4f499b250" exitCode=0 Nov 03 16:48:14.623732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.623252 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx" Nov 03 16:48:14.623732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.623282 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx" event={"ID":"e46654f6-5985-4318-98e0-81e5f85b6a9e","Type":"ContainerDied","Data":"d3cd3c1e87f2d4f131ad2737debea088d22e22f200faef3d789349f4f499b250"} Nov 03 16:48:14.623732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.623339 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx" event={"ID":"e46654f6-5985-4318-98e0-81e5f85b6a9e","Type":"ContainerDied","Data":"8dc9bbabef12c3a3f1974df90de692d27232fae4e54a42845806fa3c3a591893"} Nov 03 16:48:14.623732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.623368 4947 scope.go:122] "RemoveContainer" containerID="d3cd3c1e87f2d4f131ad2737debea088d22e22f200faef3d789349f4f499b250" Nov 03 16:48:14.625035 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.625012 4947 generic.go:356] "Generic (PLEG): container finished" podID="e08739cb-9aea-48c7-ab97-6d57c629880c" containerID="a1713c5ed697f35e9e06d762ec9c5cd84194630dc31de2bfc0ead4a917c5981f" exitCode=0 Nov 03 16:48:14.625150 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.625079 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" Nov 03 16:48:14.625150 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.625124 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" event={"ID":"e08739cb-9aea-48c7-ab97-6d57c629880c","Type":"ContainerDied","Data":"a1713c5ed697f35e9e06d762ec9c5cd84194630dc31de2bfc0ead4a917c5981f"} Nov 03 16:48:14.625252 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.625165 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj" event={"ID":"e08739cb-9aea-48c7-ab97-6d57c629880c","Type":"ContainerDied","Data":"af8429d15f03a5a837a19c443d5651714834024f19220830b83fe9e5509653cf"} Nov 03 16:48:14.626946 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.626921 4947 generic.go:356] "Generic (PLEG): container finished" podID="ea8adc48-00d3-4ed1-9499-77673978fad7" containerID="1b213e09135ca6e00cbbd1a6518704b5f18713f9a12168559a83f7db59137f35" exitCode=0 Nov 03 16:48:14.627074 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.626950 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8" event={"ID":"ea8adc48-00d3-4ed1-9499-77673978fad7","Type":"ContainerDied","Data":"1b213e09135ca6e00cbbd1a6518704b5f18713f9a12168559a83f7db59137f35"} Nov 03 16:48:14.627074 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.626998 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8" Nov 03 16:48:14.627074 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.627020 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8" event={"ID":"ea8adc48-00d3-4ed1-9499-77673978fad7","Type":"ContainerDied","Data":"e478d768268d3ceae18c72b73ddc75290eb63c13cb5abb5969ac266585fb1276"} Nov 03 16:48:14.638471 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.638445 4947 scope.go:122] "RemoveContainer" containerID="d3cd3c1e87f2d4f131ad2737debea088d22e22f200faef3d789349f4f499b250" Nov 03 16:48:14.638997 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:48:14.638975 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3cd3c1e87f2d4f131ad2737debea088d22e22f200faef3d789349f4f499b250\": container with ID starting with d3cd3c1e87f2d4f131ad2737debea088d22e22f200faef3d789349f4f499b250 not found: ID does not exist" containerID="d3cd3c1e87f2d4f131ad2737debea088d22e22f200faef3d789349f4f499b250" Nov 03 16:48:14.639104 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.639006 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3cd3c1e87f2d4f131ad2737debea088d22e22f200faef3d789349f4f499b250"} err="failed to get container status \"d3cd3c1e87f2d4f131ad2737debea088d22e22f200faef3d789349f4f499b250\": rpc error: code = NotFound desc = could not find container \"d3cd3c1e87f2d4f131ad2737debea088d22e22f200faef3d789349f4f499b250\": container with ID starting with d3cd3c1e87f2d4f131ad2737debea088d22e22f200faef3d789349f4f499b250 not found: ID does not exist" Nov 03 16:48:14.639104 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.639025 4947 scope.go:122] "RemoveContainer" containerID="a1713c5ed697f35e9e06d762ec9c5cd84194630dc31de2bfc0ead4a917c5981f" Nov 03 16:48:14.648566 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.648550 4947 scope.go:122] "RemoveContainer" containerID="a1713c5ed697f35e9e06d762ec9c5cd84194630dc31de2bfc0ead4a917c5981f" Nov 03 16:48:14.648883 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:48:14.648861 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1713c5ed697f35e9e06d762ec9c5cd84194630dc31de2bfc0ead4a917c5981f\": container with ID starting with a1713c5ed697f35e9e06d762ec9c5cd84194630dc31de2bfc0ead4a917c5981f not found: ID does not exist" containerID="a1713c5ed697f35e9e06d762ec9c5cd84194630dc31de2bfc0ead4a917c5981f" Nov 03 16:48:14.648961 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.648892 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1713c5ed697f35e9e06d762ec9c5cd84194630dc31de2bfc0ead4a917c5981f"} err="failed to get container status \"a1713c5ed697f35e9e06d762ec9c5cd84194630dc31de2bfc0ead4a917c5981f\": rpc error: code = NotFound desc = could not find container \"a1713c5ed697f35e9e06d762ec9c5cd84194630dc31de2bfc0ead4a917c5981f\": container with ID starting with a1713c5ed697f35e9e06d762ec9c5cd84194630dc31de2bfc0ead4a917c5981f not found: ID does not exist" Nov 03 16:48:14.648961 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.648909 4947 scope.go:122] "RemoveContainer" containerID="1b213e09135ca6e00cbbd1a6518704b5f18713f9a12168559a83f7db59137f35" Nov 03 16:48:14.661053 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.661003 4947 scope.go:122] "RemoveContainer" containerID="1b213e09135ca6e00cbbd1a6518704b5f18713f9a12168559a83f7db59137f35" Nov 03 16:48:14.672783 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:48:14.661518 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b213e09135ca6e00cbbd1a6518704b5f18713f9a12168559a83f7db59137f35\": container with ID starting with 1b213e09135ca6e00cbbd1a6518704b5f18713f9a12168559a83f7db59137f35 not found: ID does not exist" containerID="1b213e09135ca6e00cbbd1a6518704b5f18713f9a12168559a83f7db59137f35" Nov 03 16:48:14.672783 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.661565 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b213e09135ca6e00cbbd1a6518704b5f18713f9a12168559a83f7db59137f35"} err="failed to get container status \"1b213e09135ca6e00cbbd1a6518704b5f18713f9a12168559a83f7db59137f35\": rpc error: code = NotFound desc = could not find container \"1b213e09135ca6e00cbbd1a6518704b5f18713f9a12168559a83f7db59137f35\": container with ID starting with 1b213e09135ca6e00cbbd1a6518704b5f18713f9a12168559a83f7db59137f35 not found: ID does not exist" Nov 03 16:48:14.716788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.698924 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj"] Nov 03 16:48:14.716788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.699751 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgnrf\" (UniqueName: \"kubernetes.io/projected/f6c68b79-535a-403e-aab9-5a847fd988e8-kube-api-access-mgnrf\") pod \"maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq\" (UID: \"f6c68b79-535a-403e-aab9-5a847fd988e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" Nov 03 16:48:14.716788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.699796 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6c68b79-535a-403e-aab9-5a847fd988e8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq\" (UID: \"f6c68b79-535a-403e-aab9-5a847fd988e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" Nov 03 16:48:14.716788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.700005 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2cb1bb-6d8b489fb8-mp7zj"] Nov 03 16:48:14.716788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.700214 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6c68b79-535a-403e-aab9-5a847fd988e8-metrics-certs\") pod \"maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq\" (UID: \"f6c68b79-535a-403e-aab9-5a847fd988e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" Nov 03 16:48:14.716788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.702260 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c68b79-535a-403e-aab9-5a847fd988e8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq\" (UID: \"f6c68b79-535a-403e-aab9-5a847fd988e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" Nov 03 16:48:14.716788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.703220 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c68b79-535a-403e-aab9-5a847fd988e8-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq\" (UID: \"f6c68b79-535a-403e-aab9-5a847fd988e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" Nov 03 16:48:14.716788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.711821 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6c68b79-535a-403e-aab9-5a847fd988e8-metrics-certs\") pod \"maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq\" (UID: \"f6c68b79-535a-403e-aab9-5a847fd988e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" Nov 03 16:48:14.737388 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.737352 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6c68b79-535a-403e-aab9-5a847fd988e8-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq\" (UID: \"f6c68b79-535a-403e-aab9-5a847fd988e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" Nov 03 16:48:14.742536 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.742465 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgnrf\" (UniqueName: \"kubernetes.io/projected/f6c68b79-535a-403e-aab9-5a847fd988e8-kube-api-access-mgnrf\") pod \"maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq\" (UID: \"f6c68b79-535a-403e-aab9-5a847fd988e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" Nov 03 16:48:14.763869 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.763745 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8"] Nov 03 16:48:14.764308 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.764284 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2cb1bb-66d6f7d466-knns8"] Nov 03 16:48:14.784688 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.784644 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx"] Nov 03 16:48:14.791003 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.790944 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2cb1bb-6786564976-bcshx"] Nov 03 16:48:14.803168 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:14.803123 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" Nov 03 16:48:15.115529 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:48:15.115500 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c68b79_535a_403e_aab9_5a847fd988e8.slice/crio-60d1800f56d8fe8713960566457b301a3639aec597f171847bf79ef9fba33ca2 WatchSource:0}: Error finding container 60d1800f56d8fe8713960566457b301a3639aec597f171847bf79ef9fba33ca2: Status 404 returned error can't find the container with id 60d1800f56d8fe8713960566457b301a3639aec597f171847bf79ef9fba33ca2 Nov 03 16:48:15.115649 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:15.115543 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq"] Nov 03 16:48:15.694028 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:15.691922 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" event={"ID":"f6c68b79-535a-403e-aab9-5a847fd988e8","Type":"ContainerStarted","Data":"21fb8a0f4df5ca8c9015f6e7857c5c7b66ea14bdff04a5b5ea71ca2e1c2b6212"} Nov 03 16:48:15.694028 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:15.691987 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" event={"ID":"f6c68b79-535a-403e-aab9-5a847fd988e8","Type":"ContainerStarted","Data":"60d1800f56d8fe8713960566457b301a3639aec597f171847bf79ef9fba33ca2"} Nov 03 16:48:15.694028 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:15.693149 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" Nov 03 16:48:15.748680 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:15.748627 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" podStartSLOduration=1.377471181 podStartE2EDuration="1.748607427s" totalImagesPullingTime="371.136246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:15.746823028 +0000 UTC m=+2039.335822999" watchObservedRunningTime="2026-11-03 16:48:15.748607427 +0000 UTC m=+2039.337607397" Nov 03 16:48:16.587618 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:16.586828 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e08739cb-9aea-48c7-ab97-6d57c629880c" path="/var/lib/kubelet/pods/e08739cb-9aea-48c7-ab97-6d57c629880c/volumes" Nov 03 16:48:16.587618 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:16.587540 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e46654f6-5985-4318-98e0-81e5f85b6a9e" path="/var/lib/kubelet/pods/e46654f6-5985-4318-98e0-81e5f85b6a9e/volumes" Nov 03 16:48:16.588169 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:16.588146 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea8adc48-00d3-4ed1-9499-77673978fad7" path="/var/lib/kubelet/pods/ea8adc48-00d3-4ed1-9499-77673978fad7/volumes" Nov 03 16:48:22.716872 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:22.716838 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" Nov 03 16:48:24.402640 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:24.402604 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm"] Nov 03 16:48:24.403303 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:24.402914 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" podUID="3757c5e0-0c52-4c99-a008-d6d944b33983" containerName="istio-proxy" containerID="cri-o://8a3813abb3b2394d2e43c34e8ea2815c8fe440429e0d45b94beb940f220288d1" gracePeriod=30 Nov 03 16:48:25.534598 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:25.534219 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:48:25.534598 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:25.534285 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" podUID="3757c5e0-0c52-4c99-a008-d6d944b33983" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:48:26.935767 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:26.935728 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k"] Nov 03 16:48:26.943607 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:26.942976 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:26.965325 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:26.965292 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k"] Nov 03 16:48:27.024050 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.024018 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-5a7d95-openshift-default-dockercfg-p98q8\"" Nov 03 16:48:27.024320 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.024149 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-workload-socket\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.024320 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.024199 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8npbh\" (UniqueName: \"kubernetes.io/projected/a707604b-dc82-4c5b-bdc4-9f88027df05c-kube-api-access-8npbh\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.024461 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.024422 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-podinfo\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.024588 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.024564 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-data\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.024665 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.024650 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-credential-socket\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.024756 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.024682 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-token\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.024915 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.024895 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-envoy\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.025010 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.024969 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-workload-certs\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.025059 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.025031 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a707604b-dc82-4c5b-bdc4-9f88027df05c-istiod-ca-cert\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.126115 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.126075 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-data\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.126327 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.126131 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-credential-socket\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.126327 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.126153 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-token\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.126464 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.126326 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-envoy\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.126464 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.126400 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-workload-certs\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.126464 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.126438 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a707604b-dc82-4c5b-bdc4-9f88027df05c-istiod-ca-cert\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.126631 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.126492 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-workload-socket\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.126631 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.126516 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8npbh\" (UniqueName: \"kubernetes.io/projected/a707604b-dc82-4c5b-bdc4-9f88027df05c-kube-api-access-8npbh\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.126631 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.126565 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-data\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.126631 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.126575 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-credential-socket\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.126631 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.126568 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-podinfo\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.126988 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.126755 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-workload-certs\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.126988 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.126947 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-workload-socket\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.127360 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.127336 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a707604b-dc82-4c5b-bdc4-9f88027df05c-istiod-ca-cert\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.128859 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.128831 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-envoy\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.129073 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.129055 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-podinfo\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.151430 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.151404 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-token\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.151687 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.151665 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8npbh\" (UniqueName: \"kubernetes.io/projected/a707604b-dc82-4c5b-bdc4-9f88027df05c-kube-api-access-8npbh\") pod \"e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.276561 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.276277 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:27.789466 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:48:27.789434 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda707604b_dc82_4c5b_bdc4_9f88027df05c.slice/crio-622176f0200dc840994a1a2cbf5a4764e8f224d1cd150affd6b611cd712bd1ce WatchSource:0}: Error finding container 622176f0200dc840994a1a2cbf5a4764e8f224d1cd150affd6b611cd712bd1ce: Status 404 returned error can't find the container with id 622176f0200dc840994a1a2cbf5a4764e8f224d1cd150affd6b611cd712bd1ce Nov 03 16:48:27.789816 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.789793 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k"] Nov 03 16:48:27.792188 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.792159 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:48:27.792258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.792218 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:48:27.792258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.792249 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:48:27.820828 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:27.820794 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" event={"ID":"a707604b-dc82-4c5b-bdc4-9f88027df05c","Type":"ContainerStarted","Data":"622176f0200dc840994a1a2cbf5a4764e8f224d1cd150affd6b611cd712bd1ce"} Nov 03 16:48:28.828740 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:28.828668 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" event={"ID":"a707604b-dc82-4c5b-bdc4-9f88027df05c","Type":"ContainerStarted","Data":"7c27267fb1dce452385c60cb3154e80321b181d26c6d7b7e80f917d0ff8df9d5"} Nov 03 16:48:28.893911 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:28.893845 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" podStartSLOduration=2.893828763 podStartE2EDuration="2.893828763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:28.890131644 +0000 UTC m=+2052.479131613" watchObservedRunningTime="2026-11-03 16:48:28.893828763 +0000 UTC m=+2052.482828732" Nov 03 16:48:29.278571 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.278540 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:29.284232 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.284210 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:29.703747 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.703692 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:48:29.772859 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.772828 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3757c5e0-0c52-4c99-a008-d6d944b33983-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3757c5e0-0c52-4c99-a008-d6d944b33983-istiod-ca-cert\") pod \"3757c5e0-0c52-4c99-a008-d6d944b33983\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " Nov 03 16:48:29.773056 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.772892 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3757c5e0-0c52-4c99-a008-d6d944b33983-kube-api-access-zxgkv\" (UniqueName: \"kubernetes.io/projected/3757c5e0-0c52-4c99-a008-d6d944b33983-kube-api-access-zxgkv\") pod \"3757c5e0-0c52-4c99-a008-d6d944b33983\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " Nov 03 16:48:29.773056 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.772947 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-credential-socket\") pod \"3757c5e0-0c52-4c99-a008-d6d944b33983\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " Nov 03 16:48:29.773180 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.773069 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-podinfo\") pod \"3757c5e0-0c52-4c99-a008-d6d944b33983\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " Nov 03 16:48:29.773180 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.773172 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-token\" (UniqueName: \"kubernetes.io/projected/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-token\") pod \"3757c5e0-0c52-4c99-a008-d6d944b33983\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " Nov 03 16:48:29.773299 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.773187 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3757c5e0-0c52-4c99-a008-d6d944b33983-istiod-ca-cert" pod "3757c5e0-0c52-4c99-a008-d6d944b33983" (UID: "3757c5e0-0c52-4c99-a008-d6d944b33983"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:29.773400 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.773341 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-data\") pod \"3757c5e0-0c52-4c99-a008-d6d944b33983\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " Nov 03 16:48:29.773400 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.773338 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-credential-socket" pod "3757c5e0-0c52-4c99-a008-d6d944b33983" (UID: "3757c5e0-0c52-4c99-a008-d6d944b33983"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:48:29.773400 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.773384 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-workload-certs\") pod \"3757c5e0-0c52-4c99-a008-d6d944b33983\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " Nov 03 16:48:29.773559 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.773433 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-envoy\") pod \"3757c5e0-0c52-4c99-a008-d6d944b33983\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " Nov 03 16:48:29.773559 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.773515 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-workload-socket\") pod \"3757c5e0-0c52-4c99-a008-d6d944b33983\" (UID: \"3757c5e0-0c52-4c99-a008-d6d944b33983\") " Nov 03 16:48:29.773668 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.773582 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-data" pod "3757c5e0-0c52-4c99-a008-d6d944b33983" (UID: "3757c5e0-0c52-4c99-a008-d6d944b33983"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:48:29.774048 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.773780 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-workload-certs" pod "3757c5e0-0c52-4c99-a008-d6d944b33983" (UID: "3757c5e0-0c52-4c99-a008-d6d944b33983"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:48:29.774048 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.774016 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-workload-socket" pod "3757c5e0-0c52-4c99-a008-d6d944b33983" (UID: "3757c5e0-0c52-4c99-a008-d6d944b33983"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:48:29.774442 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.774421 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:29.774529 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.774448 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:29.774529 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.774462 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:29.774529 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.774475 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3757c5e0-0c52-4c99-a008-d6d944b33983-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:29.774529 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.774489 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:29.776182 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.776151 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-podinfo" pod "3757c5e0-0c52-4c99-a008-d6d944b33983" (UID: "3757c5e0-0c52-4c99-a008-d6d944b33983"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:48:29.776288 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.776228 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-token" pod "3757c5e0-0c52-4c99-a008-d6d944b33983" (UID: "3757c5e0-0c52-4c99-a008-d6d944b33983"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:29.776521 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.776494 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3757c5e0-0c52-4c99-a008-d6d944b33983-kube-api-access-zxgkv" pod "3757c5e0-0c52-4c99-a008-d6d944b33983" (UID: "3757c5e0-0c52-4c99-a008-d6d944b33983"). InnerVolumeSpecName "kube-api-access-zxgkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:29.777859 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.777834 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-envoy" pod "3757c5e0-0c52-4c99-a008-d6d944b33983" (UID: "3757c5e0-0c52-4c99-a008-d6d944b33983"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:48:29.845005 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.844921 4947 generic.go:356] "Generic (PLEG): container finished" podID="3757c5e0-0c52-4c99-a008-d6d944b33983" containerID="8a3813abb3b2394d2e43c34e8ea2815c8fe440429e0d45b94beb940f220288d1" exitCode=0 Nov 03 16:48:29.845005 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.844988 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" Nov 03 16:48:29.845476 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.845002 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" event={"ID":"3757c5e0-0c52-4c99-a008-d6d944b33983","Type":"ContainerDied","Data":"8a3813abb3b2394d2e43c34e8ea2815c8fe440429e0d45b94beb940f220288d1"} Nov 03 16:48:29.845476 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.845090 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm" event={"ID":"3757c5e0-0c52-4c99-a008-d6d944b33983","Type":"ContainerDied","Data":"3e0e2ab6d0f53745ed60e935a132fd50d2469f797cd419c38a573f72069be5e6"} Nov 03 16:48:29.845476 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.845120 4947 scope.go:122] "RemoveContainer" containerID="8a3813abb3b2394d2e43c34e8ea2815c8fe440429e0d45b94beb940f220288d1" Nov 03 16:48:29.846341 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.846300 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:29.847175 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.847153 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:48:29.856815 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.856780 4947 scope.go:122] "RemoveContainer" containerID="295adc1234281ab4b599907511e063eb871c1e040e4620f7b5d77a1f62800738" Nov 03 16:48:29.866893 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.866878 4947 scope.go:122] "RemoveContainer" containerID="8a3813abb3b2394d2e43c34e8ea2815c8fe440429e0d45b94beb940f220288d1" Nov 03 16:48:29.867335 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:48:29.867318 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a3813abb3b2394d2e43c34e8ea2815c8fe440429e0d45b94beb940f220288d1\": container with ID starting with 8a3813abb3b2394d2e43c34e8ea2815c8fe440429e0d45b94beb940f220288d1 not found: ID does not exist" containerID="8a3813abb3b2394d2e43c34e8ea2815c8fe440429e0d45b94beb940f220288d1" Nov 03 16:48:29.867431 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.867349 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3813abb3b2394d2e43c34e8ea2815c8fe440429e0d45b94beb940f220288d1"} err="failed to get container status \"8a3813abb3b2394d2e43c34e8ea2815c8fe440429e0d45b94beb940f220288d1\": rpc error: code = NotFound desc = could not find container \"8a3813abb3b2394d2e43c34e8ea2815c8fe440429e0d45b94beb940f220288d1\": container with ID starting with 8a3813abb3b2394d2e43c34e8ea2815c8fe440429e0d45b94beb940f220288d1 not found: ID does not exist" Nov 03 16:48:29.867431 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.867373 4947 scope.go:122] "RemoveContainer" containerID="295adc1234281ab4b599907511e063eb871c1e040e4620f7b5d77a1f62800738" Nov 03 16:48:29.867684 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:48:29.867661 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"295adc1234281ab4b599907511e063eb871c1e040e4620f7b5d77a1f62800738\": container with ID starting with 295adc1234281ab4b599907511e063eb871c1e040e4620f7b5d77a1f62800738 not found: ID does not exist" containerID="295adc1234281ab4b599907511e063eb871c1e040e4620f7b5d77a1f62800738" Nov 03 16:48:29.867817 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.867687 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"295adc1234281ab4b599907511e063eb871c1e040e4620f7b5d77a1f62800738"} err="failed to get container status \"295adc1234281ab4b599907511e063eb871c1e040e4620f7b5d77a1f62800738\": rpc error: code = NotFound desc = could not find container \"295adc1234281ab4b599907511e063eb871c1e040e4620f7b5d77a1f62800738\": container with ID starting with 295adc1234281ab4b599907511e063eb871c1e040e4620f7b5d77a1f62800738 not found: ID does not exist" Nov 03 16:48:29.876014 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.875985 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxgkv\" (UniqueName: \"kubernetes.io/projected/3757c5e0-0c52-4c99-a008-d6d944b33983-kube-api-access-zxgkv\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:29.876014 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.876012 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:29.876129 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.876027 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:29.876129 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.876039 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3757c5e0-0c52-4c99-a008-d6d944b33983-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:29.962523 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.962488 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm"] Nov 03 16:48:29.972742 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:29.967202 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2cb1bb-openshift-default-7d77cddd87-lnbjm"] Nov 03 16:48:30.137674 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:48:30.137590 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:48:30.567857 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:30.567826 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3757c5e0-0c52-4c99-a008-d6d944b33983" path="/var/lib/kubelet/pods/3757c5e0-0c52-4c99-a008-d6d944b33983/volumes" Nov 03 16:48:31.494555 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.494521 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:48:31.607596 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.607557 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb"] Nov 03 16:48:31.608290 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.608270 4947 cpu_manager.go:414] "containerMap: removing container" podUID="3757c5e0-0c52-4c99-a008-d6d944b33983" containerName="istio-proxy" Nov 03 16:48:31.608423 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.608292 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3757c5e0-0c52-4c99-a008-d6d944b33983" containerName="istio-proxy" Nov 03 16:48:31.608516 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.608502 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="3757c5e0-0c52-4c99-a008-d6d944b33983" containerName="istio-proxy" Nov 03 16:48:31.608568 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.608523 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="3757c5e0-0c52-4c99-a008-d6d944b33983" containerName="istio-proxy" Nov 03 16:48:31.613968 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.613947 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb" Nov 03 16:48:31.635281 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.635243 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-9160f5\"" Nov 03 16:48:31.652886 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.652851 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb"] Nov 03 16:48:31.674188 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.674156 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk"] Nov 03 16:48:31.674819 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.674805 4947 cpu_manager.go:414] "containerMap: removing container" podUID="3757c5e0-0c52-4c99-a008-d6d944b33983" containerName="istio-proxy" Nov 03 16:48:31.674876 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.674820 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3757c5e0-0c52-4c99-a008-d6d944b33983" containerName="istio-proxy" Nov 03 16:48:31.678321 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.678300 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk" Nov 03 16:48:31.702977 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.702949 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-9160f5-dockercfg-wm569\"" Nov 03 16:48:31.704108 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.703691 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk"] Nov 03 16:48:31.718500 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.718469 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cqr2\" (UniqueName: \"kubernetes.io/projected/4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c-kube-api-access-2cqr2\") pod \"payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb\" (UID: \"4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb" Nov 03 16:48:31.718847 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.718811 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb\" (UID: \"4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb" Nov 03 16:48:31.820501 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.820427 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l9mz\" (UniqueName: \"kubernetes.io/projected/51d1aac5-372c-4c1b-9a6e-035abe38e775-kube-api-access-4l9mz\") pod \"payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk\" (UID: \"51d1aac5-372c-4c1b-9a6e-035abe38e775\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk" Nov 03 16:48:31.820645 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.820582 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb\" (UID: \"4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb" Nov 03 16:48:31.820711 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.820681 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cqr2\" (UniqueName: \"kubernetes.io/projected/4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c-kube-api-access-2cqr2\") pod \"payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb\" (UID: \"4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb" Nov 03 16:48:31.820845 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.820829 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d1aac5-372c-4c1b-9a6e-035abe38e775-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk\" (UID: \"51d1aac5-372c-4c1b-9a6e-035abe38e775\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk" Nov 03 16:48:31.821296 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.821275 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb\" (UID: \"4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb" Nov 03 16:48:31.845627 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.845595 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cqr2\" (UniqueName: \"kubernetes.io/projected/4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c-kube-api-access-2cqr2\") pod \"payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb\" (UID: \"4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb" Nov 03 16:48:31.927079 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.927031 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d1aac5-372c-4c1b-9a6e-035abe38e775-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk\" (UID: \"51d1aac5-372c-4c1b-9a6e-035abe38e775\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk" Nov 03 16:48:31.946035 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.945997 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l9mz\" (UniqueName: \"kubernetes.io/projected/51d1aac5-372c-4c1b-9a6e-035abe38e775-kube-api-access-4l9mz\") pod \"payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk\" (UID: \"51d1aac5-372c-4c1b-9a6e-035abe38e775\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk" Nov 03 16:48:31.946814 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.937997 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb" Nov 03 16:48:31.955747 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:31.932598 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d1aac5-372c-4c1b-9a6e-035abe38e775-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk\" (UID: \"51d1aac5-372c-4c1b-9a6e-035abe38e775\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk" Nov 03 16:48:32.007136 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:32.006807 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l9mz\" (UniqueName: \"kubernetes.io/projected/51d1aac5-372c-4c1b-9a6e-035abe38e775-kube-api-access-4l9mz\") pod \"payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk\" (UID: \"51d1aac5-372c-4c1b-9a6e-035abe38e775\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk" Nov 03 16:48:32.414156 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:32.375075 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk" Nov 03 16:48:32.911849 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:32.911811 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb"] Nov 03 16:48:32.917049 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:32.917019 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk"] Nov 03 16:48:32.985770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:32.985732 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb" event={"ID":"4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c","Type":"ContainerStarted","Data":"abd41a1920420f7dbce67ded070998d1f14c223d2ee4cc20f5ac0cbb9741e880"} Nov 03 16:48:32.986806 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:32.986782 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk" event={"ID":"51d1aac5-372c-4c1b-9a6e-035abe38e775","Type":"ContainerStarted","Data":"ab046b8e0b07a0ef34e7acf27865ccf8dce741f446557741bdb1ece313fd8a91"} Nov 03 16:48:33.974327 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:33.974288 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95"] Nov 03 16:48:33.974797 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:33.974515 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95" podUID="86c014d5-5c96-4f07-b77e-40aed86389ce" containerName="payload-pre-processing" containerID="cri-o://2b644ace6825f836166aa27f94a7bf80f01c2cb2d40c4afcb59dda9d326ce775" gracePeriod=30 Nov 03 16:48:34.037555 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:34.037520 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk" event={"ID":"51d1aac5-372c-4c1b-9a6e-035abe38e775","Type":"ContainerStarted","Data":"330c76be156249faf0a56dbf00fdf5e280206cda455f0688b77bdf42beb80f0c"} Nov 03 16:48:34.037840 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:34.037820 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk" Nov 03 16:48:34.039920 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:34.039893 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb" event={"ID":"4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c","Type":"ContainerStarted","Data":"aa07d7233d17e4e3d3b3b46bb47d86c7a7bc5be289046e842026a65887b6480b"} Nov 03 16:48:34.040451 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:34.040429 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb" Nov 03 16:48:34.090349 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:34.090282 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk" podStartSLOduration=3.090259693 podStartE2EDuration="3.090259693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:34.085410384 +0000 UTC m=+2057.674410353" watchObservedRunningTime="2026-11-03 16:48:34.090259693 +0000 UTC m=+2057.679259663" Nov 03 16:48:34.163740 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:34.161804 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb" podStartSLOduration=3.161784255 podStartE2EDuration="3.161784255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:34.160026878 +0000 UTC m=+2057.749026847" watchObservedRunningTime="2026-11-03 16:48:34.161784255 +0000 UTC m=+2057.750784224" Nov 03 16:48:34.379869 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:34.377647 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95" Nov 03 16:48:34.480016 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:34.473554 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86c014d5-5c96-4f07-b77e-40aed86389ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c014d5-5c96-4f07-b77e-40aed86389ce-plugins-config-volume\") pod \"86c014d5-5c96-4f07-b77e-40aed86389ce\" (UID: \"86c014d5-5c96-4f07-b77e-40aed86389ce\") " Nov 03 16:48:34.480016 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:34.478933 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c014d5-5c96-4f07-b77e-40aed86389ce-kube-api-access-68zq2\" (UniqueName: \"kubernetes.io/projected/86c014d5-5c96-4f07-b77e-40aed86389ce-kube-api-access-68zq2\") pod \"86c014d5-5c96-4f07-b77e-40aed86389ce\" (UID: \"86c014d5-5c96-4f07-b77e-40aed86389ce\") " Nov 03 16:48:34.499424 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:34.499387 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c014d5-5c96-4f07-b77e-40aed86389ce-plugins-config-volume" pod "86c014d5-5c96-4f07-b77e-40aed86389ce" (UID: "86c014d5-5c96-4f07-b77e-40aed86389ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:48:34.511501 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:34.511432 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c014d5-5c96-4f07-b77e-40aed86389ce-kube-api-access-68zq2" pod "86c014d5-5c96-4f07-b77e-40aed86389ce" (UID: "86c014d5-5c96-4f07-b77e-40aed86389ce"). InnerVolumeSpecName "kube-api-access-68zq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:48:34.646516 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:34.646417 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c014d5-5c96-4f07-b77e-40aed86389ce-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:34.646782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:34.646757 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68zq2\" (UniqueName: \"kubernetes.io/projected/86c014d5-5c96-4f07-b77e-40aed86389ce-kube-api-access-68zq2\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:48:35.047672 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:35.047642 4947 generic.go:356] "Generic (PLEG): container finished" podID="86c014d5-5c96-4f07-b77e-40aed86389ce" containerID="2b644ace6825f836166aa27f94a7bf80f01c2cb2d40c4afcb59dda9d326ce775" exitCode=0 Nov 03 16:48:35.048163 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:35.047684 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95" Nov 03 16:48:35.048163 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:35.047733 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95" event={"ID":"86c014d5-5c96-4f07-b77e-40aed86389ce","Type":"ContainerDied","Data":"2b644ace6825f836166aa27f94a7bf80f01c2cb2d40c4afcb59dda9d326ce775"} Nov 03 16:48:35.048163 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:35.047788 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95" event={"ID":"86c014d5-5c96-4f07-b77e-40aed86389ce","Type":"ContainerDied","Data":"b70d3c4fe783abc1056f33eb4ca05d9027fd635f73baed82ea202c15288eaadd"} Nov 03 16:48:35.048163 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:35.047817 4947 scope.go:122] "RemoveContainer" containerID="2b644ace6825f836166aa27f94a7bf80f01c2cb2d40c4afcb59dda9d326ce775" Nov 03 16:48:35.060091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:35.060072 4947 scope.go:122] "RemoveContainer" containerID="2b644ace6825f836166aa27f94a7bf80f01c2cb2d40c4afcb59dda9d326ce775" Nov 03 16:48:35.060449 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:48:35.060425 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b644ace6825f836166aa27f94a7bf80f01c2cb2d40c4afcb59dda9d326ce775\": container with ID starting with 2b644ace6825f836166aa27f94a7bf80f01c2cb2d40c4afcb59dda9d326ce775 not found: ID does not exist" containerID="2b644ace6825f836166aa27f94a7bf80f01c2cb2d40c4afcb59dda9d326ce775" Nov 03 16:48:35.060582 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:35.060458 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b644ace6825f836166aa27f94a7bf80f01c2cb2d40c4afcb59dda9d326ce775"} err="failed to get container status \"2b644ace6825f836166aa27f94a7bf80f01c2cb2d40c4afcb59dda9d326ce775\": rpc error: code = NotFound desc = could not find container \"2b644ace6825f836166aa27f94a7bf80f01c2cb2d40c4afcb59dda9d326ce775\": container with ID starting with 2b644ace6825f836166aa27f94a7bf80f01c2cb2d40c4afcb59dda9d326ce775 not found: ID does not exist" Nov 03 16:48:35.124605 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:35.124564 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95"] Nov 03 16:48:35.130678 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:35.130646 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e0a112-55488bfdfd-s2h95"] Nov 03 16:48:36.631724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:36.629089 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86c014d5-5c96-4f07-b77e-40aed86389ce" path="/var/lib/kubelet/pods/86c014d5-5c96-4f07-b77e-40aed86389ce/volumes" Nov 03 16:48:43.105705 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:43.105614 4947 generic.go:356] "Generic (PLEG): container finished" podID="bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" containerID="eca4fd1e96600bd71ebe9180b7e077788b19a865d43dfb3ad3e86d9d2813f414" exitCode=0 Nov 03 16:48:43.106108 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:43.105686 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" event={"ID":"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159","Type":"ContainerDied","Data":"eca4fd1e96600bd71ebe9180b7e077788b19a865d43dfb3ad3e86d9d2813f414"} Nov 03 16:48:43.106171 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:43.106156 4947 scope.go:122] "RemoveContainer" containerID="eca4fd1e96600bd71ebe9180b7e077788b19a865d43dfb3ad3e86d9d2813f414" Nov 03 16:48:43.107779 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:43.107746 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:48:43.107876 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:43.107836 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:48:43.107914 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:43.107875 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:48:44.154730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:44.148075 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" event={"ID":"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159","Type":"ContainerStarted","Data":"45109a19cfcbe438a052388c0126bf372081960d733e286e6d6969eecdd1b703"} Nov 03 16:48:44.362660 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:44.362623 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:48:44.363813 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:44.363789 4947 patch_prober.go:29] interesting pod/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 16:48:44.363882 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:44.363854 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" podUID="bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 16:48:45.051758 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:45.051729 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk" Nov 03 16:48:45.051917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:45.051810 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb" Nov 03 16:48:45.367073 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:45.366435 4947 patch_prober.go:29] interesting pod/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 16:48:45.367073 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:45.366503 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" podUID="bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 16:48:46.363719 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:46.363097 4947 patch_prober.go:29] interesting pod/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 16:48:46.363719 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:46.363160 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" podUID="bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 16:48:47.363099 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:47.363069 4947 patch_prober.go:29] interesting pod/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 16:48:47.363516 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:47.363119 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" podUID="bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 16:48:48.319756 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:48.319115 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:48:48.365723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:48.363070 4947 patch_prober.go:29] interesting pod/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 16:48:48.365723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:48.363128 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" podUID="bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 16:48:49.362987 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:49.362948 4947 patch_prober.go:29] interesting pod/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Nov 03 16:48:49.363171 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:49.363017 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" podUID="bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Nov 03 16:48:49.662957 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:49.662850 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg"] Nov 03 16:48:49.663691 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:49.663670 4947 cpu_manager.go:414] "containerMap: removing container" podUID="86c014d5-5c96-4f07-b77e-40aed86389ce" containerName="payload-pre-processing" Nov 03 16:48:49.663801 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:49.663710 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c014d5-5c96-4f07-b77e-40aed86389ce" containerName="payload-pre-processing" Nov 03 16:48:49.663933 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:49.663919 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c014d5-5c96-4f07-b77e-40aed86389ce" containerName="payload-pre-processing" Nov 03 16:48:49.667209 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:49.667186 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg" Nov 03 16:48:49.683432 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:49.683403 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d6bb42\"" Nov 03 16:48:49.696744 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:49.696686 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d6bb42-dockercfg-vrfkg\"" Nov 03 16:48:49.702082 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:49.702047 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg"] Nov 03 16:48:49.896723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:49.891334 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqczp\" (UniqueName: \"kubernetes.io/projected/51572f38-36e8-475b-9171-31a55d83e984-kube-api-access-zqczp\") pod \"payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg\" (UID: \"51572f38-36e8-475b-9171-31a55d83e984\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg" Nov 03 16:48:49.896723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:49.891479 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51572f38-36e8-475b-9171-31a55d83e984-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg\" (UID: \"51572f38-36e8-475b-9171-31a55d83e984\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg" Nov 03 16:48:50.008723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:49.997184 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqczp\" (UniqueName: \"kubernetes.io/projected/51572f38-36e8-475b-9171-31a55d83e984-kube-api-access-zqczp\") pod \"payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg\" (UID: \"51572f38-36e8-475b-9171-31a55d83e984\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg" Nov 03 16:48:50.008723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:49.997326 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51572f38-36e8-475b-9171-31a55d83e984-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg\" (UID: \"51572f38-36e8-475b-9171-31a55d83e984\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg" Nov 03 16:48:50.008723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:49.998416 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51572f38-36e8-475b-9171-31a55d83e984-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg\" (UID: \"51572f38-36e8-475b-9171-31a55d83e984\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg" Nov 03 16:48:50.058059 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:50.058030 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqczp\" (UniqueName: \"kubernetes.io/projected/51572f38-36e8-475b-9171-31a55d83e984-kube-api-access-zqczp\") pod \"payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg\" (UID: \"51572f38-36e8-475b-9171-31a55d83e984\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg" Nov 03 16:48:50.280835 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:50.280752 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg" Nov 03 16:48:50.407466 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:50.406583 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:48:50.413544 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:50.413467 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:48:50.591279 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:48:50.590614 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51572f38_36e8_475b_9171_31a55d83e984.slice/crio-328e1285e14b575e8f404b747b34b0d568a0354f349f65ddeb6a46803f2e01ab WatchSource:0}: Error finding container 328e1285e14b575e8f404b747b34b0d568a0354f349f65ddeb6a46803f2e01ab: Status 404 returned error can't find the container with id 328e1285e14b575e8f404b747b34b0d568a0354f349f65ddeb6a46803f2e01ab Nov 03 16:48:50.591279 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:50.591035 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg"] Nov 03 16:48:51.262386 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:51.262355 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg" event={"ID":"51572f38-36e8-475b-9171-31a55d83e984","Type":"ContainerStarted","Data":"93aacb597f3187d50d97fbeacfe19981fb785433236070237d251d5ddaf18b39"} Nov 03 16:48:51.262857 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:51.262408 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg" event={"ID":"51572f38-36e8-475b-9171-31a55d83e984","Type":"ContainerStarted","Data":"328e1285e14b575e8f404b747b34b0d568a0354f349f65ddeb6a46803f2e01ab"} Nov 03 16:48:51.262857 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:51.262527 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg" Nov 03 16:48:51.306805 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:51.306751 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg" podStartSLOduration=2.306736822 podStartE2EDuration="2.306736822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:48:51.305647121 +0000 UTC m=+2074.894647104" watchObservedRunningTime="2026-11-03 16:48:51.306736822 +0000 UTC m=+2074.895736788" Nov 03 16:48:56.497980 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:48:56.497949 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:49:00.132500 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:49:00.132454 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:49:02.273383 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:02.273354 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg" Nov 03 16:49:12.715190 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:12.715152 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f"] Nov 03 16:49:12.720049 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:12.720025 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" Nov 03 16:49:12.737969 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:12.737901 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-d6bb42-serving-cert\"" Nov 03 16:49:12.743407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:12.743379 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f"] Nov 03 16:49:12.868307 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:12.868274 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71ad661-ebdc-4bce-b870-a113baeba8e8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f\" (UID: \"e71ad661-ebdc-4bce-b870-a113baeba8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" Nov 03 16:49:12.868555 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:12.868531 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e71ad661-ebdc-4bce-b870-a113baeba8e8-metrics-certs\") pod \"maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f\" (UID: \"e71ad661-ebdc-4bce-b870-a113baeba8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" Nov 03 16:49:12.868637 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:12.868622 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e71ad661-ebdc-4bce-b870-a113baeba8e8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f\" (UID: \"e71ad661-ebdc-4bce-b870-a113baeba8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" Nov 03 16:49:12.868688 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:12.868665 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d5fl\" (UniqueName: \"kubernetes.io/projected/e71ad661-ebdc-4bce-b870-a113baeba8e8-kube-api-access-5d5fl\") pod \"maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f\" (UID: \"e71ad661-ebdc-4bce-b870-a113baeba8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" Nov 03 16:49:12.973903 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:12.973809 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71ad661-ebdc-4bce-b870-a113baeba8e8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f\" (UID: \"e71ad661-ebdc-4bce-b870-a113baeba8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" Nov 03 16:49:12.974908 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:12.974876 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e71ad661-ebdc-4bce-b870-a113baeba8e8-metrics-certs\") pod \"maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f\" (UID: \"e71ad661-ebdc-4bce-b870-a113baeba8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" Nov 03 16:49:12.975194 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:12.975175 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e71ad661-ebdc-4bce-b870-a113baeba8e8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f\" (UID: \"e71ad661-ebdc-4bce-b870-a113baeba8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" Nov 03 16:49:12.975308 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:12.975243 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d5fl\" (UniqueName: \"kubernetes.io/projected/e71ad661-ebdc-4bce-b870-a113baeba8e8-kube-api-access-5d5fl\") pod \"maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f\" (UID: \"e71ad661-ebdc-4bce-b870-a113baeba8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" Nov 03 16:49:12.976969 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:12.976945 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71ad661-ebdc-4bce-b870-a113baeba8e8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f\" (UID: \"e71ad661-ebdc-4bce-b870-a113baeba8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" Nov 03 16:49:12.982024 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:12.981993 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e71ad661-ebdc-4bce-b870-a113baeba8e8-metrics-certs\") pod \"maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f\" (UID: \"e71ad661-ebdc-4bce-b870-a113baeba8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" Nov 03 16:49:13.000724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:12.999631 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e71ad661-ebdc-4bce-b870-a113baeba8e8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f\" (UID: \"e71ad661-ebdc-4bce-b870-a113baeba8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" Nov 03 16:49:13.010674 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:13.010467 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d5fl\" (UniqueName: \"kubernetes.io/projected/e71ad661-ebdc-4bce-b870-a113baeba8e8-kube-api-access-5d5fl\") pod \"maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f\" (UID: \"e71ad661-ebdc-4bce-b870-a113baeba8e8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" Nov 03 16:49:13.033589 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:13.033514 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" Nov 03 16:49:13.343205 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:49:13.343162 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71ad661_ebdc_4bce_b870_a113baeba8e8.slice/crio-1bdab06a8ee44dd63cd8d8a42af382f44b16b43ed3adf22f32c31d3253b6845c WatchSource:0}: Error finding container 1bdab06a8ee44dd63cd8d8a42af382f44b16b43ed3adf22f32c31d3253b6845c: Status 404 returned error can't find the container with id 1bdab06a8ee44dd63cd8d8a42af382f44b16b43ed3adf22f32c31d3253b6845c Nov 03 16:49:13.343433 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:13.343414 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f"] Nov 03 16:49:13.344676 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:13.344660 4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:49:13.468019 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:13.467986 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" event={"ID":"e71ad661-ebdc-4bce-b870-a113baeba8e8","Type":"ContainerStarted","Data":"1bdab06a8ee44dd63cd8d8a42af382f44b16b43ed3adf22f32c31d3253b6845c"} Nov 03 16:49:14.476033 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:14.475997 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" event={"ID":"e71ad661-ebdc-4bce-b870-a113baeba8e8","Type":"ContainerStarted","Data":"6eaa23c448af139b2f310860fa6ba9ca9323024d669d3e8c8db34045f1f4e5a6"} Nov 03 16:49:14.476517 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:14.476194 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" Nov 03 16:49:14.523644 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:14.523590 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" podStartSLOduration=2.241957077 podStartE2EDuration="2.523575191s" totalImagesPullingTime="281.618114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:14.52133841 +0000 UTC m=+2098.110338378" watchObservedRunningTime="2026-11-03 16:49:14.523575191 +0000 UTC m=+2098.112575158" Nov 03 16:49:16.837627 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:16.837602 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:49:16.838868 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:16.838844 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:49:17.191146 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:17.191065 4947 scope.go:122] "RemoveContainer" containerID="e57574ecb44e8301d8ebc7a32f4ec6b72183c13eaba3ee3fedccb5c073d11fe1" Nov 03 16:49:20.487614 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:20.487587 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" Nov 03 16:49:20.662796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:20.662096 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t"] Nov 03 16:49:20.662796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:20.662737 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" podUID="95645580-4766-498e-80f5-26586a5c8b18" containerName="istio-proxy" containerID="cri-o://08f7ae79b9b8dc1f6e3c3cf5710ac6b78aa94956a701968d4f61b4b9101b9825" gracePeriod=30 Nov 03 16:49:24.756301 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:24.756261 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:49:24.756861 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:24.756341 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" podUID="95645580-4766-498e-80f5-26586a5c8b18" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:49:25.956207 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:25.956171 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:49:26.116722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.115414 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-workload-socket\") pod \"95645580-4766-498e-80f5-26586a5c8b18\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " Nov 03 16:49:26.116722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.115479 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95645580-4766-498e-80f5-26586a5c8b18-kube-api-access-ndqr7\" (UniqueName: \"kubernetes.io/projected/95645580-4766-498e-80f5-26586a5c8b18-kube-api-access-ndqr7\") pod \"95645580-4766-498e-80f5-26586a5c8b18\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " Nov 03 16:49:26.116722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.115632 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95645580-4766-498e-80f5-26586a5c8b18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95645580-4766-498e-80f5-26586a5c8b18-istiod-ca-cert\") pod \"95645580-4766-498e-80f5-26586a5c8b18\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " Nov 03 16:49:26.116722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.115836 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-credential-socket\") pod \"95645580-4766-498e-80f5-26586a5c8b18\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " Nov 03 16:49:26.116722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.115897 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-workload-certs\") pod \"95645580-4766-498e-80f5-26586a5c8b18\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " Nov 03 16:49:26.116722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.115967 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-istio-envoy\") pod \"95645580-4766-498e-80f5-26586a5c8b18\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " Nov 03 16:49:26.116722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.116001 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95645580-4766-498e-80f5-26586a5c8b18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95645580-4766-498e-80f5-26586a5c8b18-istio-podinfo\") pod \"95645580-4766-498e-80f5-26586a5c8b18\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " Nov 03 16:49:26.116722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.116032 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-istio-data\") pod \"95645580-4766-498e-80f5-26586a5c8b18\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " Nov 03 16:49:26.116722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.116123 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95645580-4766-498e-80f5-26586a5c8b18-istio-token\" (UniqueName: \"kubernetes.io/projected/95645580-4766-498e-80f5-26586a5c8b18-istio-token\") pod \"95645580-4766-498e-80f5-26586a5c8b18\" (UID: \"95645580-4766-498e-80f5-26586a5c8b18\") " Nov 03 16:49:26.163924 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.160267 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-credential-socket" pod "95645580-4766-498e-80f5-26586a5c8b18" (UID: "95645580-4766-498e-80f5-26586a5c8b18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:26.163924 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.160497 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-workload-socket" pod "95645580-4766-498e-80f5-26586a5c8b18" (UID: "95645580-4766-498e-80f5-26586a5c8b18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:26.166878 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.166443 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-workload-certs" pod "95645580-4766-498e-80f5-26586a5c8b18" (UID: "95645580-4766-498e-80f5-26586a5c8b18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:26.167728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.167315 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-istio-data" pod "95645580-4766-498e-80f5-26586a5c8b18" (UID: "95645580-4766-498e-80f5-26586a5c8b18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:26.168216 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.168192 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95645580-4766-498e-80f5-26586a5c8b18-istiod-ca-cert" pod "95645580-4766-498e-80f5-26586a5c8b18" (UID: "95645580-4766-498e-80f5-26586a5c8b18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:26.169231 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.169207 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95645580-4766-498e-80f5-26586a5c8b18-kube-api-access-ndqr7" pod "95645580-4766-498e-80f5-26586a5c8b18" (UID: "95645580-4766-498e-80f5-26586a5c8b18"). InnerVolumeSpecName "kube-api-access-ndqr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:26.170150 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.170111 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95645580-4766-498e-80f5-26586a5c8b18-istio-podinfo" pod "95645580-4766-498e-80f5-26586a5c8b18" (UID: "95645580-4766-498e-80f5-26586a5c8b18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:49:26.171428 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.171401 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-istio-envoy" pod "95645580-4766-498e-80f5-26586a5c8b18" (UID: "95645580-4766-498e-80f5-26586a5c8b18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:26.171951 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.171913 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95645580-4766-498e-80f5-26586a5c8b18-istio-token" pod "95645580-4766-498e-80f5-26586a5c8b18" (UID: "95645580-4766-498e-80f5-26586a5c8b18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:26.218988 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.218904 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:26.218988 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.218937 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:26.218988 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.218948 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95645580-4766-498e-80f5-26586a5c8b18-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:26.218988 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.218959 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:26.218988 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.218970 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95645580-4766-498e-80f5-26586a5c8b18-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:26.218988 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.218980 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:26.219318 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.218992 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndqr7\" (UniqueName: \"kubernetes.io/projected/95645580-4766-498e-80f5-26586a5c8b18-kube-api-access-ndqr7\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:26.219318 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.219006 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95645580-4766-498e-80f5-26586a5c8b18-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:26.219318 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.219029 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95645580-4766-498e-80f5-26586a5c8b18-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:26.594887 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.594849 4947 generic.go:356] "Generic (PLEG): container finished" podID="95645580-4766-498e-80f5-26586a5c8b18" containerID="08f7ae79b9b8dc1f6e3c3cf5710ac6b78aa94956a701968d4f61b4b9101b9825" exitCode=0 Nov 03 16:49:26.595069 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.594916 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" Nov 03 16:49:26.595069 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.594928 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" event={"ID":"95645580-4766-498e-80f5-26586a5c8b18","Type":"ContainerDied","Data":"08f7ae79b9b8dc1f6e3c3cf5710ac6b78aa94956a701968d4f61b4b9101b9825"} Nov 03 16:49:26.595069 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.594987 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t" event={"ID":"95645580-4766-498e-80f5-26586a5c8b18","Type":"ContainerDied","Data":"0faccdbe58f4fdb2e810a3517c2c4093c9ba0bd99602ec4c1aecf50d04eaddca"} Nov 03 16:49:26.595069 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.595014 4947 scope.go:122] "RemoveContainer" containerID="08f7ae79b9b8dc1f6e3c3cf5710ac6b78aa94956a701968d4f61b4b9101b9825" Nov 03 16:49:26.605120 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.605097 4947 scope.go:122] "RemoveContainer" containerID="b3902df6880a5f0ca22df435066c9e4a6613218565d2281a21361b0c50fb3ccb" Nov 03 16:49:26.615154 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.615126 4947 scope.go:122] "RemoveContainer" containerID="08f7ae79b9b8dc1f6e3c3cf5710ac6b78aa94956a701968d4f61b4b9101b9825" Nov 03 16:49:26.619027 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:49:26.619000 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f7ae79b9b8dc1f6e3c3cf5710ac6b78aa94956a701968d4f61b4b9101b9825\": container with ID starting with 08f7ae79b9b8dc1f6e3c3cf5710ac6b78aa94956a701968d4f61b4b9101b9825 not found: ID does not exist" containerID="08f7ae79b9b8dc1f6e3c3cf5710ac6b78aa94956a701968d4f61b4b9101b9825" Nov 03 16:49:26.619159 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.619039 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f7ae79b9b8dc1f6e3c3cf5710ac6b78aa94956a701968d4f61b4b9101b9825"} err="failed to get container status \"08f7ae79b9b8dc1f6e3c3cf5710ac6b78aa94956a701968d4f61b4b9101b9825\": rpc error: code = NotFound desc = could not find container \"08f7ae79b9b8dc1f6e3c3cf5710ac6b78aa94956a701968d4f61b4b9101b9825\": container with ID starting with 08f7ae79b9b8dc1f6e3c3cf5710ac6b78aa94956a701968d4f61b4b9101b9825 not found: ID does not exist" Nov 03 16:49:26.619159 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.619059 4947 scope.go:122] "RemoveContainer" containerID="b3902df6880a5f0ca22df435066c9e4a6613218565d2281a21361b0c50fb3ccb" Nov 03 16:49:26.619375 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:49:26.619357 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3902df6880a5f0ca22df435066c9e4a6613218565d2281a21361b0c50fb3ccb\": container with ID starting with b3902df6880a5f0ca22df435066c9e4a6613218565d2281a21361b0c50fb3ccb not found: ID does not exist" containerID="b3902df6880a5f0ca22df435066c9e4a6613218565d2281a21361b0c50fb3ccb" Nov 03 16:49:26.619429 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.619391 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3902df6880a5f0ca22df435066c9e4a6613218565d2281a21361b0c50fb3ccb"} err="failed to get container status \"b3902df6880a5f0ca22df435066c9e4a6613218565d2281a21361b0c50fb3ccb\": rpc error: code = NotFound desc = could not find container \"b3902df6880a5f0ca22df435066c9e4a6613218565d2281a21361b0c50fb3ccb\": container with ID starting with b3902df6880a5f0ca22df435066c9e4a6613218565d2281a21361b0c50fb3ccb not found: ID does not exist" Nov 03 16:49:26.718965 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.718927 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t"] Nov 03 16:49:26.732245 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.732209 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-624649-openshift-default-8bc87cb9c-q298t"] Nov 03 16:49:26.824185 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.824146 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2"] Nov 03 16:49:26.824473 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:26.824441 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" podUID="64bfff21-904a-40fb-829a-7f49e05f0d09" containerName="istio-proxy" containerID="cri-o://ed569fa9ca2a514b8f42ddc72260708f37149706674c344b1f3e2ded95895f15" gracePeriod=30 Nov 03 16:49:27.603846 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:27.603816 4947 generic.go:356] "Generic (PLEG): container finished" podID="a707604b-dc82-4c5b-bdc4-9f88027df05c" containerID="7c27267fb1dce452385c60cb3154e80321b181d26c6d7b7e80f917d0ff8df9d5" exitCode=0 Nov 03 16:49:27.604384 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:27.603905 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" event={"ID":"a707604b-dc82-4c5b-bdc4-9f88027df05c","Type":"ContainerDied","Data":"7c27267fb1dce452385c60cb3154e80321b181d26c6d7b7e80f917d0ff8df9d5"} Nov 03 16:49:27.604490 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:27.604471 4947 scope.go:122] "RemoveContainer" containerID="7c27267fb1dce452385c60cb3154e80321b181d26c6d7b7e80f917d0ff8df9d5" Nov 03 16:49:27.606281 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:27.606251 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:49:27.606387 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:27.606323 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:49:27.606387 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:27.606354 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:49:28.566646 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:28.566616 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95645580-4766-498e-80f5-26586a5c8b18" path="/var/lib/kubelet/pods/95645580-4766-498e-80f5-26586a5c8b18/volumes" Nov 03 16:49:28.613425 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:28.613394 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" event={"ID":"a707604b-dc82-4c5b-bdc4-9f88027df05c","Type":"ContainerStarted","Data":"92476e59fd61af2b994259bc879bf173121d6beca551490ae843a3aa9f373e41"} Nov 03 16:49:29.279801 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:29.279759 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:49:29.280794 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:29.280767 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 16:49:29.280954 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:29.280820 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" podUID="a707604b-dc82-4c5b-bdc4-9f88027df05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 16:49:29.845916 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:29.845870 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:49:30.150403 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:49:30.150321 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:49:30.278756 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:30.278723 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 16:49:30.279321 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:30.278797 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" podUID="a707604b-dc82-4c5b-bdc4-9f88027df05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 16:49:30.882685 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:30.882642 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:49:30.883154 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:30.882718 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" podUID="64bfff21-904a-40fb-829a-7f49e05f0d09" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:49:31.278349 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:31.278314 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 16:49:31.278522 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:31.278377 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" podUID="a707604b-dc82-4c5b-bdc4-9f88027df05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 16:49:32.248092 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.248068 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:49:32.278394 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.278361 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 16:49:32.278584 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.278411 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" podUID="a707604b-dc82-4c5b-bdc4-9f88027df05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 16:49:32.353117 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.353084 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-workload-certs\") pod \"64bfff21-904a-40fb-829a-7f49e05f0d09\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " Nov 03 16:49:32.353319 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.353204 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-podinfo\") pod \"64bfff21-904a-40fb-829a-7f49e05f0d09\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " Nov 03 16:49:32.353319 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.353265 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-envoy\") pod \"64bfff21-904a-40fb-829a-7f49e05f0d09\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " Nov 03 16:49:32.353319 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.353301 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-workload-socket\") pod \"64bfff21-904a-40fb-829a-7f49e05f0d09\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " Nov 03 16:49:32.353490 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.353349 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64bfff21-904a-40fb-829a-7f49e05f0d09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64bfff21-904a-40fb-829a-7f49e05f0d09-istiod-ca-cert\") pod \"64bfff21-904a-40fb-829a-7f49e05f0d09\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " Nov 03 16:49:32.353490 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.353385 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-workload-certs" pod "64bfff21-904a-40fb-829a-7f49e05f0d09" (UID: "64bfff21-904a-40fb-829a-7f49e05f0d09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:32.353590 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.353556 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-credential-socket\") pod \"64bfff21-904a-40fb-829a-7f49e05f0d09\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " Nov 03 16:49:32.353675 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.353658 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-data\") pod \"64bfff21-904a-40fb-829a-7f49e05f0d09\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " Nov 03 16:49:32.353785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.353756 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64bfff21-904a-40fb-829a-7f49e05f0d09-istiod-ca-cert" pod "64bfff21-904a-40fb-829a-7f49e05f0d09" (UID: "64bfff21-904a-40fb-829a-7f49e05f0d09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:32.353785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.353778 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-token\" (UniqueName: \"kubernetes.io/projected/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-token\") pod \"64bfff21-904a-40fb-829a-7f49e05f0d09\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " Nov 03 16:49:32.353893 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.353841 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64bfff21-904a-40fb-829a-7f49e05f0d09-kube-api-access-dzmdx\" (UniqueName: \"kubernetes.io/projected/64bfff21-904a-40fb-829a-7f49e05f0d09-kube-api-access-dzmdx\") pod \"64bfff21-904a-40fb-829a-7f49e05f0d09\" (UID: \"64bfff21-904a-40fb-829a-7f49e05f0d09\") " Nov 03 16:49:32.354018 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.353995 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-workload-socket" pod "64bfff21-904a-40fb-829a-7f49e05f0d09" (UID: "64bfff21-904a-40fb-829a-7f49e05f0d09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:32.354209 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.354188 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-data" pod "64bfff21-904a-40fb-829a-7f49e05f0d09" (UID: "64bfff21-904a-40fb-829a-7f49e05f0d09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:32.354385 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.354363 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-credential-socket" pod "64bfff21-904a-40fb-829a-7f49e05f0d09" (UID: "64bfff21-904a-40fb-829a-7f49e05f0d09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:32.355775 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.355751 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:32.355867 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.355781 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64bfff21-904a-40fb-829a-7f49e05f0d09-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:32.355867 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.355794 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:32.355867 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.355806 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:32.355867 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.355819 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:32.358137 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.358113 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-podinfo" pod "64bfff21-904a-40fb-829a-7f49e05f0d09" (UID: "64bfff21-904a-40fb-829a-7f49e05f0d09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:49:32.358392 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.358372 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-token" pod "64bfff21-904a-40fb-829a-7f49e05f0d09" (UID: "64bfff21-904a-40fb-829a-7f49e05f0d09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:32.358541 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.358512 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-envoy" pod "64bfff21-904a-40fb-829a-7f49e05f0d09" (UID: "64bfff21-904a-40fb-829a-7f49e05f0d09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:32.360012 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.359990 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64bfff21-904a-40fb-829a-7f49e05f0d09-kube-api-access-dzmdx" pod "64bfff21-904a-40fb-829a-7f49e05f0d09" (UID: "64bfff21-904a-40fb-829a-7f49e05f0d09"). InnerVolumeSpecName "kube-api-access-dzmdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:32.457323 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.457287 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:32.457323 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.457320 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzmdx\" (UniqueName: \"kubernetes.io/projected/64bfff21-904a-40fb-829a-7f49e05f0d09-kube-api-access-dzmdx\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:32.457538 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.457337 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:32.457538 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.457350 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64bfff21-904a-40fb-829a-7f49e05f0d09-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:32.650954 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.650866 4947 generic.go:356] "Generic (PLEG): container finished" podID="64bfff21-904a-40fb-829a-7f49e05f0d09" containerID="ed569fa9ca2a514b8f42ddc72260708f37149706674c344b1f3e2ded95895f15" exitCode=0 Nov 03 16:49:32.650954 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.650913 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" Nov 03 16:49:32.651183 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.650909 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" event={"ID":"64bfff21-904a-40fb-829a-7f49e05f0d09","Type":"ContainerDied","Data":"ed569fa9ca2a514b8f42ddc72260708f37149706674c344b1f3e2ded95895f15"} Nov 03 16:49:32.651183 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.651014 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2" event={"ID":"64bfff21-904a-40fb-829a-7f49e05f0d09","Type":"ContainerDied","Data":"07cce8c7abcc2bcc98683d2978ee4cba142b0dfb332d6ffadf79de05eb8e9d86"} Nov 03 16:49:32.651183 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.651049 4947 scope.go:122] "RemoveContainer" containerID="ed569fa9ca2a514b8f42ddc72260708f37149706674c344b1f3e2ded95895f15" Nov 03 16:49:32.661755 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.661729 4947 scope.go:122] "RemoveContainer" containerID="ed569fa9ca2a514b8f42ddc72260708f37149706674c344b1f3e2ded95895f15" Nov 03 16:49:32.662101 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:49:32.662081 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed569fa9ca2a514b8f42ddc72260708f37149706674c344b1f3e2ded95895f15\": container with ID starting with ed569fa9ca2a514b8f42ddc72260708f37149706674c344b1f3e2ded95895f15 not found: ID does not exist" containerID="ed569fa9ca2a514b8f42ddc72260708f37149706674c344b1f3e2ded95895f15" Nov 03 16:49:32.662199 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.662107 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed569fa9ca2a514b8f42ddc72260708f37149706674c344b1f3e2ded95895f15"} err="failed to get container status \"ed569fa9ca2a514b8f42ddc72260708f37149706674c344b1f3e2ded95895f15\": rpc error: code = NotFound desc = could not find container \"ed569fa9ca2a514b8f42ddc72260708f37149706674c344b1f3e2ded95895f15\": container with ID starting with ed569fa9ca2a514b8f42ddc72260708f37149706674c344b1f3e2ded95895f15 not found: ID does not exist" Nov 03 16:49:32.725257 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.725217 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2"] Nov 03 16:49:32.728937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:32.728903 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-x2kq2"] Nov 03 16:49:33.278463 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:33.278429 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 16:49:33.278978 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:33.278482 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" podUID="a707604b-dc82-4c5b-bdc4-9f88027df05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 16:49:34.094588 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:34.094554 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9"] Nov 03 16:49:34.094896 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:34.094869 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" podUID="1971c181-22d1-4f72-a032-f65f2a6b765f" containerName="istio-proxy" containerID="cri-o://6419d9b304b49cef0c84c4ec3ea7f471ada98fdb8dbf9f4e023d8feb38f7c059" gracePeriod=30 Nov 03 16:49:34.278220 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:34.278184 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 16:49:34.278399 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:34.278248 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" podUID="a707604b-dc82-4c5b-bdc4-9f88027df05c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 16:49:34.566135 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:34.566102 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64bfff21-904a-40fb-829a-7f49e05f0d09" path="/var/lib/kubelet/pods/64bfff21-904a-40fb-829a-7f49e05f0d09/volumes" Nov 03 16:49:35.103726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:35.099222 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:49:35.283081 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:35.283053 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:49:35.284240 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:35.284222 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:49:36.886035 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:36.885998 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96"] Nov 03 16:49:36.886517 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:36.886299 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" podUID="bde4c160-1141-4f29-b691-a13cfa66d1f4" containerName="maas-api" containerID="cri-o://be567175c6851e161ec6220d7a093efd59bab67ce1bd0d291e5f260dd769e701" gracePeriod=30 Nov 03 16:49:37.228035 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.228011 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:49:37.325532 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.325495 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bde4c160-1141-4f29-b691-a13cfa66d1f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde4c160-1141-4f29-b691-a13cfa66d1f4-openshift-service-ca\") pod \"bde4c160-1141-4f29-b691-a13cfa66d1f4\" (UID: \"bde4c160-1141-4f29-b691-a13cfa66d1f4\") " Nov 03 16:49:37.325778 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.325551 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bde4c160-1141-4f29-b691-a13cfa66d1f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde4c160-1141-4f29-b691-a13cfa66d1f4-maas-api-tls\") pod \"bde4c160-1141-4f29-b691-a13cfa66d1f4\" (UID: \"bde4c160-1141-4f29-b691-a13cfa66d1f4\") " Nov 03 16:49:37.325778 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.325599 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde4c160-1141-4f29-b691-a13cfa66d1f4-kube-api-access-l45rf\" (UniqueName: \"kubernetes.io/projected/bde4c160-1141-4f29-b691-a13cfa66d1f4-kube-api-access-l45rf\") pod \"bde4c160-1141-4f29-b691-a13cfa66d1f4\" (UID: \"bde4c160-1141-4f29-b691-a13cfa66d1f4\") " Nov 03 16:49:37.325778 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.325741 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bde4c160-1141-4f29-b691-a13cfa66d1f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde4c160-1141-4f29-b691-a13cfa66d1f4-metrics-certs\") pod \"bde4c160-1141-4f29-b691-a13cfa66d1f4\" (UID: \"bde4c160-1141-4f29-b691-a13cfa66d1f4\") " Nov 03 16:49:37.326012 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.325951 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde4c160-1141-4f29-b691-a13cfa66d1f4-openshift-service-ca" pod "bde4c160-1141-4f29-b691-a13cfa66d1f4" (UID: "bde4c160-1141-4f29-b691-a13cfa66d1f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:37.326855 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.326830 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde4c160-1141-4f29-b691-a13cfa66d1f4-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:37.338746 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.329002 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde4c160-1141-4f29-b691-a13cfa66d1f4-kube-api-access-l45rf" pod "bde4c160-1141-4f29-b691-a13cfa66d1f4" (UID: "bde4c160-1141-4f29-b691-a13cfa66d1f4"). InnerVolumeSpecName "kube-api-access-l45rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:37.338746 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.329142 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bde4c160-1141-4f29-b691-a13cfa66d1f4-metrics-certs" pod "bde4c160-1141-4f29-b691-a13cfa66d1f4" (UID: "bde4c160-1141-4f29-b691-a13cfa66d1f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:49:37.338746 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.337771 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bde4c160-1141-4f29-b691-a13cfa66d1f4-maas-api-tls" pod "bde4c160-1141-4f29-b691-a13cfa66d1f4" (UID: "bde4c160-1141-4f29-b691-a13cfa66d1f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:49:37.478774 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.476989 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bde4c160-1141-4f29-b691-a13cfa66d1f4-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:37.478774 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.477027 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bde4c160-1141-4f29-b691-a13cfa66d1f4-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:37.478774 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.477041 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l45rf\" (UniqueName: \"kubernetes.io/projected/bde4c160-1141-4f29-b691-a13cfa66d1f4-kube-api-access-l45rf\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:37.789132 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.789100 4947 generic.go:356] "Generic (PLEG): container finished" podID="bde4c160-1141-4f29-b691-a13cfa66d1f4" containerID="be567175c6851e161ec6220d7a093efd59bab67ce1bd0d291e5f260dd769e701" exitCode=0 Nov 03 16:49:37.789322 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.789146 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" Nov 03 16:49:37.789322 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.789187 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" event={"ID":"bde4c160-1141-4f29-b691-a13cfa66d1f4","Type":"ContainerDied","Data":"be567175c6851e161ec6220d7a093efd59bab67ce1bd0d291e5f260dd769e701"} Nov 03 16:49:37.789322 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.789245 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96" event={"ID":"bde4c160-1141-4f29-b691-a13cfa66d1f4","Type":"ContainerDied","Data":"cc5cfc5c9351d45b2680df7b6aa7875316aab1bef51d8aaa2775b80fcb9bf93e"} Nov 03 16:49:37.789322 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.789274 4947 scope.go:122] "RemoveContainer" containerID="be567175c6851e161ec6220d7a093efd59bab67ce1bd0d291e5f260dd769e701" Nov 03 16:49:37.816136 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.812274 4947 scope.go:122] "RemoveContainer" containerID="be567175c6851e161ec6220d7a093efd59bab67ce1bd0d291e5f260dd769e701" Nov 03 16:49:37.823087 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:49:37.822281 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be567175c6851e161ec6220d7a093efd59bab67ce1bd0d291e5f260dd769e701\": container with ID starting with be567175c6851e161ec6220d7a093efd59bab67ce1bd0d291e5f260dd769e701 not found: ID does not exist" containerID="be567175c6851e161ec6220d7a093efd59bab67ce1bd0d291e5f260dd769e701" Nov 03 16:49:37.823087 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.822332 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be567175c6851e161ec6220d7a093efd59bab67ce1bd0d291e5f260dd769e701"} err="failed to get container status \"be567175c6851e161ec6220d7a093efd59bab67ce1bd0d291e5f260dd769e701\": rpc error: code = NotFound desc = could not find container \"be567175c6851e161ec6220d7a093efd59bab67ce1bd0d291e5f260dd769e701\": container with ID starting with be567175c6851e161ec6220d7a093efd59bab67ce1bd0d291e5f260dd769e701 not found: ID does not exist" Nov 03 16:49:37.931204 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.931168 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96"] Nov 03 16:49:37.935425 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:37.935392 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d71e12-cf75846df-87k96"] Nov 03 16:49:38.567267 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:38.567235 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bde4c160-1141-4f29-b691-a13cfa66d1f4" path="/var/lib/kubelet/pods/bde4c160-1141-4f29-b691-a13cfa66d1f4/volumes" Nov 03 16:49:39.467228 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.467204 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:49:39.558375 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.558265 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-workload-socket\") pod \"1971c181-22d1-4f72-a032-f65f2a6b765f\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " Nov 03 16:49:39.558375 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.558359 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-envoy\") pod \"1971c181-22d1-4f72-a032-f65f2a6b765f\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " Nov 03 16:49:39.558796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.558501 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-credential-socket\") pod \"1971c181-22d1-4f72-a032-f65f2a6b765f\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " Nov 03 16:49:39.558796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.558547 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-token\" (UniqueName: \"kubernetes.io/projected/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-token\") pod \"1971c181-22d1-4f72-a032-f65f2a6b765f\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " Nov 03 16:49:39.558796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.558586 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-data\") pod \"1971c181-22d1-4f72-a032-f65f2a6b765f\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " Nov 03 16:49:39.558796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.558616 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-workload-certs\") pod \"1971c181-22d1-4f72-a032-f65f2a6b765f\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " Nov 03 16:49:39.558796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.558675 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1971c181-22d1-4f72-a032-f65f2a6b765f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1971c181-22d1-4f72-a032-f65f2a6b765f-istiod-ca-cert\") pod \"1971c181-22d1-4f72-a032-f65f2a6b765f\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " Nov 03 16:49:39.558796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.558763 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-podinfo\") pod \"1971c181-22d1-4f72-a032-f65f2a6b765f\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " Nov 03 16:49:39.559137 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.558830 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1971c181-22d1-4f72-a032-f65f2a6b765f-kube-api-access-qgb4x\" (UniqueName: \"kubernetes.io/projected/1971c181-22d1-4f72-a032-f65f2a6b765f-kube-api-access-qgb4x\") pod \"1971c181-22d1-4f72-a032-f65f2a6b765f\" (UID: \"1971c181-22d1-4f72-a032-f65f2a6b765f\") " Nov 03 16:49:39.580724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.579060 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-workload-certs" pod "1971c181-22d1-4f72-a032-f65f2a6b765f" (UID: "1971c181-22d1-4f72-a032-f65f2a6b765f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:39.580724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.579259 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-data" pod "1971c181-22d1-4f72-a032-f65f2a6b765f" (UID: "1971c181-22d1-4f72-a032-f65f2a6b765f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:39.583007 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.582969 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1971c181-22d1-4f72-a032-f65f2a6b765f-istiod-ca-cert" pod "1971c181-22d1-4f72-a032-f65f2a6b765f" (UID: "1971c181-22d1-4f72-a032-f65f2a6b765f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:49:39.619755 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.610175 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-workload-socket" pod "1971c181-22d1-4f72-a032-f65f2a6b765f" (UID: "1971c181-22d1-4f72-a032-f65f2a6b765f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:39.619755 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.619671 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-credential-socket" pod "1971c181-22d1-4f72-a032-f65f2a6b765f" (UID: "1971c181-22d1-4f72-a032-f65f2a6b765f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:39.621054 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.621028 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1971c181-22d1-4f72-a032-f65f2a6b765f-kube-api-access-qgb4x" pod "1971c181-22d1-4f72-a032-f65f2a6b765f" (UID: "1971c181-22d1-4f72-a032-f65f2a6b765f"). InnerVolumeSpecName "kube-api-access-qgb4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:39.625639 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.625604 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-podinfo" pod "1971c181-22d1-4f72-a032-f65f2a6b765f" (UID: "1971c181-22d1-4f72-a032-f65f2a6b765f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:49:39.626344 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.626317 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-token" pod "1971c181-22d1-4f72-a032-f65f2a6b765f" (UID: "1971c181-22d1-4f72-a032-f65f2a6b765f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:49:39.626459 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.626395 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-envoy" pod "1971c181-22d1-4f72-a032-f65f2a6b765f" (UID: "1971c181-22d1-4f72-a032-f65f2a6b765f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:49:39.663178 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.661988 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:39.663178 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.662025 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:39.663178 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.662038 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:39.663178 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.662046 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:39.663178 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.662054 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1971c181-22d1-4f72-a032-f65f2a6b765f-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:39.663178 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.662063 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:39.663178 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.662074 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgb4x\" (UniqueName: \"kubernetes.io/projected/1971c181-22d1-4f72-a032-f65f2a6b765f-kube-api-access-qgb4x\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:39.663178 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.662087 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:39.663178 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.662099 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1971c181-22d1-4f72-a032-f65f2a6b765f-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:49:39.803900 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.803872 4947 generic.go:356] "Generic (PLEG): container finished" podID="1971c181-22d1-4f72-a032-f65f2a6b765f" containerID="6419d9b304b49cef0c84c4ec3ea7f471ada98fdb8dbf9f4e023d8feb38f7c059" exitCode=0 Nov 03 16:49:39.804089 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.803928 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" Nov 03 16:49:39.804089 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.803960 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" event={"ID":"1971c181-22d1-4f72-a032-f65f2a6b765f","Type":"ContainerDied","Data":"6419d9b304b49cef0c84c4ec3ea7f471ada98fdb8dbf9f4e023d8feb38f7c059"} Nov 03 16:49:39.804089 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.804021 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9" event={"ID":"1971c181-22d1-4f72-a032-f65f2a6b765f","Type":"ContainerDied","Data":"f482fc496cd6dd564222becd22a38e1d0b0edfaa1a6f0a45b32a50d1c5895ca6"} Nov 03 16:49:39.804089 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.804052 4947 scope.go:122] "RemoveContainer" containerID="6419d9b304b49cef0c84c4ec3ea7f471ada98fdb8dbf9f4e023d8feb38f7c059" Nov 03 16:49:39.815045 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.815020 4947 scope.go:122] "RemoveContainer" containerID="6419d9b304b49cef0c84c4ec3ea7f471ada98fdb8dbf9f4e023d8feb38f7c059" Nov 03 16:49:39.815326 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:49:39.815307 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6419d9b304b49cef0c84c4ec3ea7f471ada98fdb8dbf9f4e023d8feb38f7c059\": container with ID starting with 6419d9b304b49cef0c84c4ec3ea7f471ada98fdb8dbf9f4e023d8feb38f7c059 not found: ID does not exist" containerID="6419d9b304b49cef0c84c4ec3ea7f471ada98fdb8dbf9f4e023d8feb38f7c059" Nov 03 16:49:39.815397 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.815337 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6419d9b304b49cef0c84c4ec3ea7f471ada98fdb8dbf9f4e023d8feb38f7c059"} err="failed to get container status \"6419d9b304b49cef0c84c4ec3ea7f471ada98fdb8dbf9f4e023d8feb38f7c059\": rpc error: code = NotFound desc = could not find container \"6419d9b304b49cef0c84c4ec3ea7f471ada98fdb8dbf9f4e023d8feb38f7c059\": container with ID starting with 6419d9b304b49cef0c84c4ec3ea7f471ada98fdb8dbf9f4e023d8feb38f7c059 not found: ID does not exist" Nov 03 16:49:39.863847 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.863814 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9"] Nov 03 16:49:39.867483 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:39.867457 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-2kns9"] Nov 03 16:49:40.566880 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:40.566780 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1971c181-22d1-4f72-a032-f65f2a6b765f" path="/var/lib/kubelet/pods/1971c181-22d1-4f72-a032-f65f2a6b765f/volumes" Nov 03 16:49:47.171917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.169825 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn"] Nov 03 16:49:47.171917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.170913 4947 cpu_manager.go:414] "containerMap: removing container" podUID="bde4c160-1141-4f29-b691-a13cfa66d1f4" containerName="maas-api" Nov 03 16:49:47.171917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.170936 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde4c160-1141-4f29-b691-a13cfa66d1f4" containerName="maas-api" Nov 03 16:49:47.171917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.170966 4947 cpu_manager.go:414] "containerMap: removing container" podUID="1971c181-22d1-4f72-a032-f65f2a6b765f" containerName="istio-proxy" Nov 03 16:49:47.171917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.170974 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1971c181-22d1-4f72-a032-f65f2a6b765f" containerName="istio-proxy" Nov 03 16:49:47.171917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.171032 4947 cpu_manager.go:414] "containerMap: removing container" podUID="95645580-4766-498e-80f5-26586a5c8b18" containerName="istio-proxy" Nov 03 16:49:47.171917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.171041 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95645580-4766-498e-80f5-26586a5c8b18" containerName="istio-proxy" Nov 03 16:49:47.171917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.171064 4947 cpu_manager.go:414] "containerMap: removing container" podUID="64bfff21-904a-40fb-829a-7f49e05f0d09" containerName="istio-proxy" Nov 03 16:49:47.171917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.171071 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64bfff21-904a-40fb-829a-7f49e05f0d09" containerName="istio-proxy" Nov 03 16:49:47.171917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.171087 4947 cpu_manager.go:414] "containerMap: removing container" podUID="95645580-4766-498e-80f5-26586a5c8b18" containerName="istio-proxy" Nov 03 16:49:47.171917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.171096 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95645580-4766-498e-80f5-26586a5c8b18" containerName="istio-proxy" Nov 03 16:49:47.171917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.171252 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="1971c181-22d1-4f72-a032-f65f2a6b765f" containerName="istio-proxy" Nov 03 16:49:47.171917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.171278 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="95645580-4766-498e-80f5-26586a5c8b18" containerName="istio-proxy" Nov 03 16:49:47.171917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.171295 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="64bfff21-904a-40fb-829a-7f49e05f0d09" containerName="istio-proxy" Nov 03 16:49:47.171917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.171303 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="95645580-4766-498e-80f5-26586a5c8b18" containerName="istio-proxy" Nov 03 16:49:47.171917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.171316 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde4c160-1141-4f29-b691-a13cfa66d1f4" containerName="maas-api" Nov 03 16:49:47.193780 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.189987 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn" Nov 03 16:49:47.238893 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.216415 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn"] Nov 03 16:49:47.238893 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.217385 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5a7d95-dockercfg-jtq2g\"" Nov 03 16:49:47.268784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.256315 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5njj\" (UniqueName: \"kubernetes.io/projected/007c9802-d9bf-4cd2-a062-6da9b70c31ff-kube-api-access-h5njj\") pod \"payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn\" (UID: \"007c9802-d9bf-4cd2-a062-6da9b70c31ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn" Nov 03 16:49:47.268784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.256389 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007c9802-d9bf-4cd2-a062-6da9b70c31ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn\" (UID: \"007c9802-d9bf-4cd2-a062-6da9b70c31ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn" Nov 03 16:49:47.306221 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.306172 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5a7d95\"" Nov 03 16:49:47.314772 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.313370 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg"] Nov 03 16:49:47.327854 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.327785 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg"] Nov 03 16:49:47.328050 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.327971 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg" Nov 03 16:49:47.368551 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.368514 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5njj\" (UniqueName: \"kubernetes.io/projected/007c9802-d9bf-4cd2-a062-6da9b70c31ff-kube-api-access-h5njj\") pod \"payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn\" (UID: \"007c9802-d9bf-4cd2-a062-6da9b70c31ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn" Nov 03 16:49:47.378960 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.378924 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007c9802-d9bf-4cd2-a062-6da9b70c31ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn\" (UID: \"007c9802-d9bf-4cd2-a062-6da9b70c31ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn" Nov 03 16:49:47.380436 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.380407 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007c9802-d9bf-4cd2-a062-6da9b70c31ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn\" (UID: \"007c9802-d9bf-4cd2-a062-6da9b70c31ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn" Nov 03 16:49:47.410935 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.410764 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5njj\" (UniqueName: \"kubernetes.io/projected/007c9802-d9bf-4cd2-a062-6da9b70c31ff-kube-api-access-h5njj\") pod \"payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn\" (UID: \"007c9802-d9bf-4cd2-a062-6da9b70c31ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn" Nov 03 16:49:47.481100 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.481019 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz8xz\" (UniqueName: \"kubernetes.io/projected/ff88d741-c433-40a3-804b-24742f4a8140-kube-api-access-dz8xz\") pod \"payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg\" (UID: \"ff88d741-c433-40a3-804b-24742f4a8140\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg" Nov 03 16:49:47.481249 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.481221 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff88d741-c433-40a3-804b-24742f4a8140-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg\" (UID: \"ff88d741-c433-40a3-804b-24742f4a8140\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg" Nov 03 16:49:47.583286 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.583247 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff88d741-c433-40a3-804b-24742f4a8140-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg\" (UID: \"ff88d741-c433-40a3-804b-24742f4a8140\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg" Nov 03 16:49:47.583532 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.583513 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz8xz\" (UniqueName: \"kubernetes.io/projected/ff88d741-c433-40a3-804b-24742f4a8140-kube-api-access-dz8xz\") pod \"payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg\" (UID: \"ff88d741-c433-40a3-804b-24742f4a8140\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg" Nov 03 16:49:47.584274 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.584248 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff88d741-c433-40a3-804b-24742f4a8140-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg\" (UID: \"ff88d741-c433-40a3-804b-24742f4a8140\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg" Nov 03 16:49:47.608748 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.608715 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz8xz\" (UniqueName: \"kubernetes.io/projected/ff88d741-c433-40a3-804b-24742f4a8140-kube-api-access-dz8xz\") pod \"payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg\" (UID: \"ff88d741-c433-40a3-804b-24742f4a8140\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg" Nov 03 16:49:47.613652 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.613617 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn" Nov 03 16:49:47.664203 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:47.664160 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg" Nov 03 16:49:48.050725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:48.034212 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn"] Nov 03 16:49:48.096181 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:48.078860 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg"] Nov 03 16:49:48.096181 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:49:48.087577 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007c9802_d9bf_4cd2_a062_6da9b70c31ff.slice/crio-62ba83f9fe7e122436b64fb9ccefb9d05893b32cce0782b1320cf627abde668c WatchSource:0}: Error finding container 62ba83f9fe7e122436b64fb9ccefb9d05893b32cce0782b1320cf627abde668c: Status 404 returned error can't find the container with id 62ba83f9fe7e122436b64fb9ccefb9d05893b32cce0782b1320cf627abde668c Nov 03 16:49:49.040747 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:49.040586 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg" event={"ID":"ff88d741-c433-40a3-804b-24742f4a8140","Type":"ContainerStarted","Data":"1cd6fcdc81b7f31ddd2bc09a43fc7e388f87b3c262da66360c1b756059357b1c"} Nov 03 16:49:49.040747 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:49.040653 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg" event={"ID":"ff88d741-c433-40a3-804b-24742f4a8140","Type":"ContainerStarted","Data":"910cf57f8995852d1af60c32e4a82f17798469327202a7d5f2817e703af9538f"} Nov 03 16:49:49.040747 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:49.040680 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg" Nov 03 16:49:49.042748 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:49.042681 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn" event={"ID":"007c9802-d9bf-4cd2-a062-6da9b70c31ff","Type":"ContainerStarted","Data":"945760414b599febdff619517e8718ca91ae1eec147b4dce751b41e6f88d1ce0"} Nov 03 16:49:49.042890 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:49.042772 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn" event={"ID":"007c9802-d9bf-4cd2-a062-6da9b70c31ff","Type":"ContainerStarted","Data":"62ba83f9fe7e122436b64fb9ccefb9d05893b32cce0782b1320cf627abde668c"} Nov 03 16:49:49.043424 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:49.043405 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn" Nov 03 16:49:49.096277 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:49.096209 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg" podStartSLOduration=2.096186817 podStartE2EDuration="2.096186817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:49.091447048 +0000 UTC m=+2132.680447016" watchObservedRunningTime="2026-11-03 16:49:49.096186817 +0000 UTC m=+2132.685186786" Nov 03 16:49:49.162254 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:49:49.161031 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn" podStartSLOduration=2.161009699 podStartE2EDuration="2.161009699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:49:49.158132605 +0000 UTC m=+2132.747132574" watchObservedRunningTime="2026-11-03 16:49:49.161009699 +0000 UTC m=+2132.750009667" Nov 03 16:50:00.053928 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:00.053895 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg" Nov 03 16:50:00.136635 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:50:00.136596 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:50:01.069352 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:01.069310 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn" Nov 03 16:50:10.259454 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:10.259421 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9"] Nov 03 16:50:10.264430 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:10.264405 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:50:10.297750 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:10.297714 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9"] Nov 03 16:50:10.351239 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:10.351195 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0adbe8a-947c-433a-886e-c824cc5d0566-maas-api-tls\") pod \"maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9\" (UID: \"a0adbe8a-947c-433a-886e-c824cc5d0566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:50:10.351440 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:10.351307 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pvpl\" (UniqueName: \"kubernetes.io/projected/a0adbe8a-947c-433a-886e-c824cc5d0566-kube-api-access-5pvpl\") pod \"maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9\" (UID: \"a0adbe8a-947c-433a-886e-c824cc5d0566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:50:10.351440 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:10.351344 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0adbe8a-947c-433a-886e-c824cc5d0566-metrics-certs\") pod \"maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9\" (UID: \"a0adbe8a-947c-433a-886e-c824cc5d0566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:50:10.351571 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:10.351550 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0adbe8a-947c-433a-886e-c824cc5d0566-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9\" (UID: \"a0adbe8a-947c-433a-886e-c824cc5d0566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:50:10.376206 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:10.376177 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-223659-serving-cert\"" Nov 03 16:50:10.464771 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:10.457825 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0adbe8a-947c-433a-886e-c824cc5d0566-maas-api-tls\") pod \"maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9\" (UID: \"a0adbe8a-947c-433a-886e-c824cc5d0566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:50:10.464771 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:10.457910 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pvpl\" (UniqueName: \"kubernetes.io/projected/a0adbe8a-947c-433a-886e-c824cc5d0566-kube-api-access-5pvpl\") pod \"maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9\" (UID: \"a0adbe8a-947c-433a-886e-c824cc5d0566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:50:10.464771 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:10.457938 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0adbe8a-947c-433a-886e-c824cc5d0566-metrics-certs\") pod \"maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9\" (UID: \"a0adbe8a-947c-433a-886e-c824cc5d0566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:50:10.464771 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:10.458110 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0adbe8a-947c-433a-886e-c824cc5d0566-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9\" (UID: \"a0adbe8a-947c-433a-886e-c824cc5d0566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:50:10.464771 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:10.459172 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0adbe8a-947c-433a-886e-c824cc5d0566-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9\" (UID: \"a0adbe8a-947c-433a-886e-c824cc5d0566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:50:10.464771 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:50:10.459290 4947 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-serving-cert: secret "maas-api-e2e-praxis-a-223659-serving-cert" not found Nov 03 16:50:10.464771 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:50:10.459350 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0adbe8a-947c-433a-886e-c824cc5d0566-maas-api-tls podName:a0adbe8a-947c-433a-886e-c824cc5d0566 nodeName:}" failed. No retries permitted until 2026-11-03 16:50:10.959331904 +0000 UTC m=+2154.548331865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a0adbe8a-947c-433a-886e-c824cc5d0566-maas-api-tls") pod "maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" (UID: "a0adbe8a-947c-433a-886e-c824cc5d0566") : secret "maas-api-e2e-praxis-a-223659-serving-cert" not found Nov 03 16:50:10.480868 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:10.469121 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0adbe8a-947c-433a-886e-c824cc5d0566-metrics-certs\") pod \"maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9\" (UID: \"a0adbe8a-947c-433a-886e-c824cc5d0566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:50:10.504934 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:10.504900 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pvpl\" (UniqueName: \"kubernetes.io/projected/a0adbe8a-947c-433a-886e-c824cc5d0566-kube-api-access-5pvpl\") pod \"maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9\" (UID: \"a0adbe8a-947c-433a-886e-c824cc5d0566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:50:10.972668 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:10.972634 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0adbe8a-947c-433a-886e-c824cc5d0566-maas-api-tls\") pod \"maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9\" (UID: \"a0adbe8a-947c-433a-886e-c824cc5d0566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:50:10.975586 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:10.975551 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0adbe8a-947c-433a-886e-c824cc5d0566-maas-api-tls\") pod \"maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9\" (UID: \"a0adbe8a-947c-433a-886e-c824cc5d0566\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:50:11.181834 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:11.181795 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:50:11.567594 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:11.567563 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9"] Nov 03 16:50:11.567970 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:50:11.567681 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0adbe8a_947c_433a_886e_c824cc5d0566.slice/crio-66f602b5efc82b741089ae8e73153f26fdf54d125b7d15461bb9858a7d34c5a1 WatchSource:0}: Error finding container 66f602b5efc82b741089ae8e73153f26fdf54d125b7d15461bb9858a7d34c5a1: Status 404 returned error can't find the container with id 66f602b5efc82b741089ae8e73153f26fdf54d125b7d15461bb9858a7d34c5a1 Nov 03 16:50:11.838316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:11.838230 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7"] Nov 03 16:50:11.893174 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:11.893131 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7"] Nov 03 16:50:11.893550 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:11.893518 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7" Nov 03 16:50:11.915490 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:11.915447 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-549271\"" Nov 03 16:50:11.915847 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:11.915829 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-549271-dockercfg-d58cp\"" Nov 03 16:50:12.038778 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:12.038690 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462f73ae-b874-4e71-b21b-5793533767ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7\" (UID: \"462f73ae-b874-4e71-b21b-5793533767ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7" Nov 03 16:50:12.039347 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:12.039286 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2slxx\" (UniqueName: \"kubernetes.io/projected/462f73ae-b874-4e71-b21b-5793533767ac-kube-api-access-2slxx\") pod \"payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7\" (UID: \"462f73ae-b874-4e71-b21b-5793533767ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7" Nov 03 16:50:12.140402 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:12.140369 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2slxx\" (UniqueName: \"kubernetes.io/projected/462f73ae-b874-4e71-b21b-5793533767ac-kube-api-access-2slxx\") pod \"payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7\" (UID: \"462f73ae-b874-4e71-b21b-5793533767ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7" Nov 03 16:50:12.140581 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:12.140558 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462f73ae-b874-4e71-b21b-5793533767ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7\" (UID: \"462f73ae-b874-4e71-b21b-5793533767ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7" Nov 03 16:50:12.141375 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:12.141355 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462f73ae-b874-4e71-b21b-5793533767ac-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7\" (UID: \"462f73ae-b874-4e71-b21b-5793533767ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7" Nov 03 16:50:12.167880 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:12.167847 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2slxx\" (UniqueName: \"kubernetes.io/projected/462f73ae-b874-4e71-b21b-5793533767ac-kube-api-access-2slxx\") pod \"payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7\" (UID: \"462f73ae-b874-4e71-b21b-5793533767ac\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7" Nov 03 16:50:12.278206 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:12.278176 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7" Nov 03 16:50:12.283909 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:12.283875 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" event={"ID":"a0adbe8a-947c-433a-886e-c824cc5d0566","Type":"ContainerStarted","Data":"5c3479ed84629b7471423c5f4626cfea4ad1186c46436dbbc5faa3d084a10024"} Nov 03 16:50:12.284046 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:12.283937 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" event={"ID":"a0adbe8a-947c-433a-886e-c824cc5d0566","Type":"ContainerStarted","Data":"66f602b5efc82b741089ae8e73153f26fdf54d125b7d15461bb9858a7d34c5a1"} Nov 03 16:50:12.284046 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:12.283964 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:50:12.350288 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:12.350241 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" podStartSLOduration=1.850119688 podStartE2EDuration="2.350216192s" totalImagesPullingTime="500.096504ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:50:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:50:12.335254447 +0000 UTC m=+2155.924254410" watchObservedRunningTime="2026-11-03 16:50:12.350216192 +0000 UTC m=+2155.939216158" Nov 03 16:50:12.603267 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:12.603235 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7"] Nov 03 16:50:12.603766 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:50:12.603274 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462f73ae_b874_4e71_b21b_5793533767ac.slice/crio-dffc7376db3dd9cf308d73146239e1ff2eb41c9017e6cc42b1b22455437e6b75 WatchSource:0}: Error finding container dffc7376db3dd9cf308d73146239e1ff2eb41c9017e6cc42b1b22455437e6b75: Status 404 returned error can't find the container with id dffc7376db3dd9cf308d73146239e1ff2eb41c9017e6cc42b1b22455437e6b75 Nov 03 16:50:13.292138 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:13.292080 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7" event={"ID":"462f73ae-b874-4e71-b21b-5793533767ac","Type":"ContainerStarted","Data":"1e498a3845407d37d9f474ccc45b7db50a65ba31f1005cd8d463f63a88b6508d"} Nov 03 16:50:13.292327 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:13.292142 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7" event={"ID":"462f73ae-b874-4e71-b21b-5793533767ac","Type":"ContainerStarted","Data":"dffc7376db3dd9cf308d73146239e1ff2eb41c9017e6cc42b1b22455437e6b75"} Nov 03 16:50:13.292327 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:13.292173 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7" Nov 03 16:50:13.338408 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:13.338355 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7" podStartSLOduration=2.3383402699999998 podStartE2EDuration="2.33834027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:50:13.337041355 +0000 UTC m=+2156.926041337" watchObservedRunningTime="2026-11-03 16:50:13.33834027 +0000 UTC m=+2156.927340236" Nov 03 16:50:14.419583 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:14.419553 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:50:18.297665 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:18.297631 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:50:24.353723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:24.353678 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7" Nov 03 16:50:30.131747 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:50:30.131691 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:50:32.886267 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:32.886229 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq"] Nov 03 16:50:32.891987 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:32.891954 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" Nov 03 16:50:32.913775 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:32.913666 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq"] Nov 03 16:50:32.918463 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:32.918221 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-549271-serving-cert\"" Nov 03 16:50:33.000000 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:32.999961 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdc8h\" (UniqueName: \"kubernetes.io/projected/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-kube-api-access-fdc8h\") pod \"maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq\" (UID: \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" Nov 03 16:50:33.000201 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:33.000136 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-metrics-certs\") pod \"maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq\" (UID: \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" Nov 03 16:50:33.000201 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:33.000172 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq\" (UID: \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" Nov 03 16:50:33.000422 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:33.000403 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq\" (UID: \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" Nov 03 16:50:33.101283 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:33.101243 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdc8h\" (UniqueName: \"kubernetes.io/projected/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-kube-api-access-fdc8h\") pod \"maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq\" (UID: \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" Nov 03 16:50:33.101488 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:33.101373 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-metrics-certs\") pod \"maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq\" (UID: \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" Nov 03 16:50:33.101488 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:33.101404 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq\" (UID: \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" Nov 03 16:50:33.101609 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:33.101590 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq\" (UID: \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" Nov 03 16:50:33.102232 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:33.102162 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq\" (UID: \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" Nov 03 16:50:33.104084 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:33.104061 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq\" (UID: \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" Nov 03 16:50:33.104184 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:33.104098 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-metrics-certs\") pod \"maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq\" (UID: \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" Nov 03 16:50:33.126253 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:33.126219 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdc8h\" (UniqueName: \"kubernetes.io/projected/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-kube-api-access-fdc8h\") pod \"maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq\" (UID: \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" Nov 03 16:50:33.206131 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:33.206036 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" Nov 03 16:50:33.563271 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:33.563237 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq"] Nov 03 16:50:33.563454 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:50:33.563433 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3275cc93_2a98_42d2_9f4a_84ce9c10fbe5.slice/crio-29932b659759dec87f3157bd5efce409529001c2d33f681b2f1f3f84f887adca WatchSource:0}: Error finding container 29932b659759dec87f3157bd5efce409529001c2d33f681b2f1f3f84f887adca: Status 404 returned error can't find the container with id 29932b659759dec87f3157bd5efce409529001c2d33f681b2f1f3f84f887adca Nov 03 16:50:34.537407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:34.537338 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" event={"ID":"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5","Type":"ContainerStarted","Data":"8727ebbe07cce33501979af62f9149a7476e3751d3bbc469750335af2ae7f92a"} Nov 03 16:50:34.537946 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:34.537424 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" Nov 03 16:50:34.537946 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:34.537456 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" event={"ID":"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5","Type":"ContainerStarted","Data":"29932b659759dec87f3157bd5efce409529001c2d33f681b2f1f3f84f887adca"} Nov 03 16:50:34.584463 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:34.584411 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" podStartSLOduration=2.247491061 podStartE2EDuration="2.584396816s" totalImagesPullingTime="336.905755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:50:34.582847559 +0000 UTC m=+2178.171847541" watchObservedRunningTime="2026-11-03 16:50:34.584396816 +0000 UTC m=+2178.173396784" Nov 03 16:50:40.549387 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:40.549355 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" Nov 03 16:50:51.721035 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:51.720999 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g"] Nov 03 16:50:51.726756 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:51.726736 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g" Nov 03 16:50:51.749351 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:51.749324 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f2923b-dockercfg-frp8t\"" Nov 03 16:50:51.749351 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:51.749327 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f2923b\"" Nov 03 16:50:51.768238 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:51.768207 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g"] Nov 03 16:50:51.852734 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:51.852459 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvsj2\" (UniqueName: \"kubernetes.io/projected/40a2532d-b751-47c7-b625-dc293b948e01-kube-api-access-mvsj2\") pod \"payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g\" (UID: \"40a2532d-b751-47c7-b625-dc293b948e01\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g" Nov 03 16:50:51.852734 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:51.852638 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40a2532d-b751-47c7-b625-dc293b948e01-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g\" (UID: \"40a2532d-b751-47c7-b625-dc293b948e01\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g" Nov 03 16:50:51.956265 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:51.956216 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40a2532d-b751-47c7-b625-dc293b948e01-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g\" (UID: \"40a2532d-b751-47c7-b625-dc293b948e01\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g" Nov 03 16:50:51.977777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:51.974142 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvsj2\" (UniqueName: \"kubernetes.io/projected/40a2532d-b751-47c7-b625-dc293b948e01-kube-api-access-mvsj2\") pod \"payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g\" (UID: \"40a2532d-b751-47c7-b625-dc293b948e01\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g" Nov 03 16:50:51.989849 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:51.957371 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40a2532d-b751-47c7-b625-dc293b948e01-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g\" (UID: \"40a2532d-b751-47c7-b625-dc293b948e01\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g" Nov 03 16:50:52.054857 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:52.054824 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvsj2\" (UniqueName: \"kubernetes.io/projected/40a2532d-b751-47c7-b625-dc293b948e01-kube-api-access-mvsj2\") pod \"payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g\" (UID: \"40a2532d-b751-47c7-b625-dc293b948e01\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g" Nov 03 16:50:52.064419 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:52.064373 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g" Nov 03 16:50:52.381496 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:50:52.381462 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a2532d_b751_47c7_b625_dc293b948e01.slice/crio-8b95ade4d266dc32210d63b63f26abeeb9163204cf128b1b3e0d779761c7597e WatchSource:0}: Error finding container 8b95ade4d266dc32210d63b63f26abeeb9163204cf128b1b3e0d779761c7597e: Status 404 returned error can't find the container with id 8b95ade4d266dc32210d63b63f26abeeb9163204cf128b1b3e0d779761c7597e Nov 03 16:50:52.382745 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:52.382718 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g"] Nov 03 16:50:52.780823 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:52.780790 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g" event={"ID":"40a2532d-b751-47c7-b625-dc293b948e01","Type":"ContainerStarted","Data":"43226cecce37df2b028413de3c4fb2b2c5d28782b83ab38212967ff126e032a2"} Nov 03 16:50:52.781263 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:52.780843 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g" event={"ID":"40a2532d-b751-47c7-b625-dc293b948e01","Type":"ContainerStarted","Data":"8b95ade4d266dc32210d63b63f26abeeb9163204cf128b1b3e0d779761c7597e"} Nov 03 16:50:52.781263 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:52.780885 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g" Nov 03 16:50:52.829030 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:52.828965 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g" podStartSLOduration=1.828950203 podStartE2EDuration="1.828950203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:50:52.824685579 +0000 UTC m=+2196.413685546" watchObservedRunningTime="2026-11-03 16:50:52.828950203 +0000 UTC m=+2196.417950169" Nov 03 16:50:53.841736 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:53.841681 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg"] Nov 03 16:50:55.535870 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:55.535834 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:50:58.842065 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:58.841978 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg" podUID="51572f38-36e8-475b-9171-31a55d83e984" containerName="payload-pre-processing" containerID="cri-o://93aacb597f3187d50d97fbeacfe19981fb785433236070237d251d5ddaf18b39" gracePeriod=25 Nov 03 16:50:59.126754 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:59.126721 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg" Nov 03 16:50:59.225917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:59.225878 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51572f38-36e8-475b-9171-31a55d83e984-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51572f38-36e8-475b-9171-31a55d83e984-plugins-config-volume\") pod \"51572f38-36e8-475b-9171-31a55d83e984\" (UID: \"51572f38-36e8-475b-9171-31a55d83e984\") " Nov 03 16:50:59.226119 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:59.226097 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51572f38-36e8-475b-9171-31a55d83e984-kube-api-access-zqczp\" (UniqueName: \"kubernetes.io/projected/51572f38-36e8-475b-9171-31a55d83e984-kube-api-access-zqczp\") pod \"51572f38-36e8-475b-9171-31a55d83e984\" (UID: \"51572f38-36e8-475b-9171-31a55d83e984\") " Nov 03 16:50:59.226326 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:59.226303 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51572f38-36e8-475b-9171-31a55d83e984-plugins-config-volume" pod "51572f38-36e8-475b-9171-31a55d83e984" (UID: "51572f38-36e8-475b-9171-31a55d83e984"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:50:59.226616 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:59.226593 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51572f38-36e8-475b-9171-31a55d83e984-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:50:59.228469 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:59.228438 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51572f38-36e8-475b-9171-31a55d83e984-kube-api-access-zqczp" pod "51572f38-36e8-475b-9171-31a55d83e984" (UID: "51572f38-36e8-475b-9171-31a55d83e984"). InnerVolumeSpecName "kube-api-access-zqczp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:50:59.327797 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:59.327760 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqczp\" (UniqueName: \"kubernetes.io/projected/51572f38-36e8-475b-9171-31a55d83e984-kube-api-access-zqczp\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:50:59.885317 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:59.885280 4947 generic.go:356] "Generic (PLEG): container finished" podID="51572f38-36e8-475b-9171-31a55d83e984" containerID="93aacb597f3187d50d97fbeacfe19981fb785433236070237d251d5ddaf18b39" exitCode=0 Nov 03 16:50:59.885768 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:59.885330 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg" Nov 03 16:50:59.885768 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:59.885365 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg" event={"ID":"51572f38-36e8-475b-9171-31a55d83e984","Type":"ContainerDied","Data":"93aacb597f3187d50d97fbeacfe19981fb785433236070237d251d5ddaf18b39"} Nov 03 16:50:59.885768 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:59.885423 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg" event={"ID":"51572f38-36e8-475b-9171-31a55d83e984","Type":"ContainerDied","Data":"328e1285e14b575e8f404b747b34b0d568a0354f349f65ddeb6a46803f2e01ab"} Nov 03 16:50:59.885768 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:59.885446 4947 scope.go:122] "RemoveContainer" containerID="93aacb597f3187d50d97fbeacfe19981fb785433236070237d251d5ddaf18b39" Nov 03 16:50:59.896127 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:59.896106 4947 scope.go:122] "RemoveContainer" containerID="93aacb597f3187d50d97fbeacfe19981fb785433236070237d251d5ddaf18b39" Nov 03 16:50:59.896430 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:50:59.896410 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93aacb597f3187d50d97fbeacfe19981fb785433236070237d251d5ddaf18b39\": container with ID starting with 93aacb597f3187d50d97fbeacfe19981fb785433236070237d251d5ddaf18b39 not found: ID does not exist" containerID="93aacb597f3187d50d97fbeacfe19981fb785433236070237d251d5ddaf18b39" Nov 03 16:50:59.896505 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:59.896443 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93aacb597f3187d50d97fbeacfe19981fb785433236070237d251d5ddaf18b39"} err="failed to get container status \"93aacb597f3187d50d97fbeacfe19981fb785433236070237d251d5ddaf18b39\": rpc error: code = NotFound desc = could not find container \"93aacb597f3187d50d97fbeacfe19981fb785433236070237d251d5ddaf18b39\": container with ID starting with 93aacb597f3187d50d97fbeacfe19981fb785433236070237d251d5ddaf18b39 not found: ID does not exist" Nov 03 16:50:59.953730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:59.953662 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg"] Nov 03 16:50:59.959030 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:50:59.959002 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d6bb42-7cc4c8ff79-mdmmg"] Nov 03 16:51:00.131620 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:51:00.131584 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:51:00.566121 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:00.566088 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51572f38-36e8-475b-9171-31a55d83e984" path="/var/lib/kubelet/pods/51572f38-36e8-475b-9171-31a55d83e984/volumes" Nov 03 16:51:03.791526 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:03.791497 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g" Nov 03 16:51:05.385838 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:05.385801 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7"] Nov 03 16:51:08.382419 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:08.382383 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq"] Nov 03 16:51:08.382857 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:08.382605 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" podUID="38262c13-420b-43af-bb5e-1cb3c62e44b0" containerName="istio-proxy" containerID="cri-o://35845104c931053556b35962ec1885befcf0bdb46424a452dbeb7efa97837bf3" gracePeriod=30 Nov 03 16:51:10.386317 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:10.386240 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7" podUID="462f73ae-b874-4e71-b21b-5793533767ac" containerName="payload-processing" containerID="cri-o://1e498a3845407d37d9f474ccc45b7db50a65ba31f1005cd8d463f63a88b6508d" gracePeriod=25 Nov 03 16:51:10.656793 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:10.656727 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7" Nov 03 16:51:10.714233 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:10.714198 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/462f73ae-b874-4e71-b21b-5793533767ac-kube-api-access-2slxx\" (UniqueName: \"kubernetes.io/projected/462f73ae-b874-4e71-b21b-5793533767ac-kube-api-access-2slxx\") pod \"462f73ae-b874-4e71-b21b-5793533767ac\" (UID: \"462f73ae-b874-4e71-b21b-5793533767ac\") " Nov 03 16:51:10.714441 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:10.714420 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/462f73ae-b874-4e71-b21b-5793533767ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462f73ae-b874-4e71-b21b-5793533767ac-plugins-config-volume\") pod \"462f73ae-b874-4e71-b21b-5793533767ac\" (UID: \"462f73ae-b874-4e71-b21b-5793533767ac\") " Nov 03 16:51:10.715001 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:10.714969 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/462f73ae-b874-4e71-b21b-5793533767ac-plugins-config-volume" pod "462f73ae-b874-4e71-b21b-5793533767ac" (UID: "462f73ae-b874-4e71-b21b-5793533767ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:10.717010 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:10.716985 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/462f73ae-b874-4e71-b21b-5793533767ac-kube-api-access-2slxx" pod "462f73ae-b874-4e71-b21b-5793533767ac" (UID: "462f73ae-b874-4e71-b21b-5793533767ac"). InnerVolumeSpecName "kube-api-access-2slxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:10.816451 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:10.816417 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2slxx\" (UniqueName: \"kubernetes.io/projected/462f73ae-b874-4e71-b21b-5793533767ac-kube-api-access-2slxx\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:10.816451 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:10.816448 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/462f73ae-b874-4e71-b21b-5793533767ac-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:10.967056 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:10.966954 4947 generic.go:356] "Generic (PLEG): container finished" podID="462f73ae-b874-4e71-b21b-5793533767ac" containerID="1e498a3845407d37d9f474ccc45b7db50a65ba31f1005cd8d463f63a88b6508d" exitCode=0 Nov 03 16:51:10.967056 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:10.966992 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7" event={"ID":"462f73ae-b874-4e71-b21b-5793533767ac","Type":"ContainerDied","Data":"1e498a3845407d37d9f474ccc45b7db50a65ba31f1005cd8d463f63a88b6508d"} Nov 03 16:51:10.967056 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:10.967015 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7" Nov 03 16:51:10.967320 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:10.967078 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7" event={"ID":"462f73ae-b874-4e71-b21b-5793533767ac","Type":"ContainerDied","Data":"dffc7376db3dd9cf308d73146239e1ff2eb41c9017e6cc42b1b22455437e6b75"} Nov 03 16:51:10.967320 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:10.967107 4947 scope.go:122] "RemoveContainer" containerID="1e498a3845407d37d9f474ccc45b7db50a65ba31f1005cd8d463f63a88b6508d" Nov 03 16:51:10.978634 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:10.978614 4947 scope.go:122] "RemoveContainer" containerID="1e498a3845407d37d9f474ccc45b7db50a65ba31f1005cd8d463f63a88b6508d" Nov 03 16:51:10.978953 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:51:10.978933 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e498a3845407d37d9f474ccc45b7db50a65ba31f1005cd8d463f63a88b6508d\": container with ID starting with 1e498a3845407d37d9f474ccc45b7db50a65ba31f1005cd8d463f63a88b6508d not found: ID does not exist" containerID="1e498a3845407d37d9f474ccc45b7db50a65ba31f1005cd8d463f63a88b6508d" Nov 03 16:51:10.979019 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:10.978962 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e498a3845407d37d9f474ccc45b7db50a65ba31f1005cd8d463f63a88b6508d"} err="failed to get container status \"1e498a3845407d37d9f474ccc45b7db50a65ba31f1005cd8d463f63a88b6508d\": rpc error: code = NotFound desc = could not find container \"1e498a3845407d37d9f474ccc45b7db50a65ba31f1005cd8d463f63a88b6508d\": container with ID starting with 1e498a3845407d37d9f474ccc45b7db50a65ba31f1005cd8d463f63a88b6508d not found: ID does not exist" Nov 03 16:51:11.023770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:11.023741 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7"] Nov 03 16:51:11.036141 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:11.036112 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-549271-5d5c54d867-2mqr7"] Nov 03 16:51:12.567193 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:12.567163 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="462f73ae-b874-4e71-b21b-5793533767ac" path="/var/lib/kubelet/pods/462f73ae-b874-4e71-b21b-5793533767ac/volumes" Nov 03 16:51:13.651834 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.651775 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:51:13.771452 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.771413 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-token\" (UniqueName: \"kubernetes.io/projected/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-token\") pod \"38262c13-420b-43af-bb5e-1cb3c62e44b0\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " Nov 03 16:51:13.771728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.771503 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-workload-certs\") pod \"38262c13-420b-43af-bb5e-1cb3c62e44b0\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " Nov 03 16:51:13.771728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.771559 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38262c13-420b-43af-bb5e-1cb3c62e44b0-kube-api-access-rxccp\" (UniqueName: \"kubernetes.io/projected/38262c13-420b-43af-bb5e-1cb3c62e44b0-kube-api-access-rxccp\") pod \"38262c13-420b-43af-bb5e-1cb3c62e44b0\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " Nov 03 16:51:13.771728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.771648 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38262c13-420b-43af-bb5e-1cb3c62e44b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38262c13-420b-43af-bb5e-1cb3c62e44b0-istiod-ca-cert\") pod \"38262c13-420b-43af-bb5e-1cb3c62e44b0\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " Nov 03 16:51:13.771728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.771684 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-data\") pod \"38262c13-420b-43af-bb5e-1cb3c62e44b0\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " Nov 03 16:51:13.771988 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.771776 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-podinfo\") pod \"38262c13-420b-43af-bb5e-1cb3c62e44b0\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " Nov 03 16:51:13.771988 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.771819 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-envoy\") pod \"38262c13-420b-43af-bb5e-1cb3c62e44b0\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " Nov 03 16:51:13.771988 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.771953 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-workload-socket\") pod \"38262c13-420b-43af-bb5e-1cb3c62e44b0\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " Nov 03 16:51:13.772150 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.771997 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-credential-socket\") pod \"38262c13-420b-43af-bb5e-1cb3c62e44b0\" (UID: \"38262c13-420b-43af-bb5e-1cb3c62e44b0\") " Nov 03 16:51:13.772210 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.772162 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-data" pod "38262c13-420b-43af-bb5e-1cb3c62e44b0" (UID: "38262c13-420b-43af-bb5e-1cb3c62e44b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:13.772684 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.772431 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-workload-socket" pod "38262c13-420b-43af-bb5e-1cb3c62e44b0" (UID: "38262c13-420b-43af-bb5e-1cb3c62e44b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:13.772684 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.772658 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-credential-socket" pod "38262c13-420b-43af-bb5e-1cb3c62e44b0" (UID: "38262c13-420b-43af-bb5e-1cb3c62e44b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:13.773172 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.772790 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:13.773172 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.772809 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:13.773172 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.772821 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:13.773526 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.773497 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38262c13-420b-43af-bb5e-1cb3c62e44b0-istiod-ca-cert" pod "38262c13-420b-43af-bb5e-1cb3c62e44b0" (UID: "38262c13-420b-43af-bb5e-1cb3c62e44b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:13.773969 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.773945 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-workload-certs" pod "38262c13-420b-43af-bb5e-1cb3c62e44b0" (UID: "38262c13-420b-43af-bb5e-1cb3c62e44b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:13.774932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.774899 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-podinfo" pod "38262c13-420b-43af-bb5e-1cb3c62e44b0" (UID: "38262c13-420b-43af-bb5e-1cb3c62e44b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:51:13.776190 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.776166 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38262c13-420b-43af-bb5e-1cb3c62e44b0-kube-api-access-rxccp" pod "38262c13-420b-43af-bb5e-1cb3c62e44b0" (UID: "38262c13-420b-43af-bb5e-1cb3c62e44b0"). InnerVolumeSpecName "kube-api-access-rxccp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:13.782283 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.782245 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-token" pod "38262c13-420b-43af-bb5e-1cb3c62e44b0" (UID: "38262c13-420b-43af-bb5e-1cb3c62e44b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:13.783489 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.783467 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-envoy" pod "38262c13-420b-43af-bb5e-1cb3c62e44b0" (UID: "38262c13-420b-43af-bb5e-1cb3c62e44b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:13.874105 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.874060 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:13.874105 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.874095 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxccp\" (UniqueName: \"kubernetes.io/projected/38262c13-420b-43af-bb5e-1cb3c62e44b0-kube-api-access-rxccp\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:13.874105 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.874110 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38262c13-420b-43af-bb5e-1cb3c62e44b0-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:13.874342 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.874124 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:13.874342 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.874134 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:13.874342 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.874145 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38262c13-420b-43af-bb5e-1cb3c62e44b0-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:13.989040 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.988958 4947 generic.go:356] "Generic (PLEG): container finished" podID="38262c13-420b-43af-bb5e-1cb3c62e44b0" containerID="35845104c931053556b35962ec1885befcf0bdb46424a452dbeb7efa97837bf3" exitCode=0 Nov 03 16:51:13.989040 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.989013 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" Nov 03 16:51:13.989223 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.989050 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" event={"ID":"38262c13-420b-43af-bb5e-1cb3c62e44b0","Type":"ContainerDied","Data":"35845104c931053556b35962ec1885befcf0bdb46424a452dbeb7efa97837bf3"} Nov 03 16:51:13.989223 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.989113 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq" event={"ID":"38262c13-420b-43af-bb5e-1cb3c62e44b0","Type":"ContainerDied","Data":"208a99156ffbf8f1fa1ade0dc9baa4075e22911053eaf6d264952a1b5485c1a5"} Nov 03 16:51:13.989223 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.989146 4947 scope.go:122] "RemoveContainer" containerID="35845104c931053556b35962ec1885befcf0bdb46424a452dbeb7efa97837bf3" Nov 03 16:51:13.999768 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:13.999750 4947 scope.go:122] "RemoveContainer" containerID="35845104c931053556b35962ec1885befcf0bdb46424a452dbeb7efa97837bf3" Nov 03 16:51:14.000071 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:51:14.000049 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35845104c931053556b35962ec1885befcf0bdb46424a452dbeb7efa97837bf3\": container with ID starting with 35845104c931053556b35962ec1885befcf0bdb46424a452dbeb7efa97837bf3 not found: ID does not exist" containerID="35845104c931053556b35962ec1885befcf0bdb46424a452dbeb7efa97837bf3" Nov 03 16:51:14.000130 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:14.000083 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35845104c931053556b35962ec1885befcf0bdb46424a452dbeb7efa97837bf3"} err="failed to get container status \"35845104c931053556b35962ec1885befcf0bdb46424a452dbeb7efa97837bf3\": rpc error: code = NotFound desc = could not find container \"35845104c931053556b35962ec1885befcf0bdb46424a452dbeb7efa97837bf3\": container with ID starting with 35845104c931053556b35962ec1885befcf0bdb46424a452dbeb7efa97837bf3 not found: ID does not exist" Nov 03 16:51:14.046014 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:14.045982 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq"] Nov 03 16:51:14.058960 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:14.058930 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-9rqsq"] Nov 03 16:51:14.565801 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:14.565771 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38262c13-420b-43af-bb5e-1cb3c62e44b0" path="/var/lib/kubelet/pods/38262c13-420b-43af-bb5e-1cb3c62e44b0/volumes" Nov 03 16:51:16.964615 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:16.964577 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5"] Nov 03 16:51:16.965256 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:16.965219 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" podUID="8c8da48a-94af-4d09-aef7-21c1a69092c7" containerName="istio-proxy" containerID="cri-o://4e7780711b3da942e5e1abad8f7b2e0a3bb094e22119043e2282a9398ed212ea" gracePeriod=30 Nov 03 16:51:17.666724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:17.666429 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755"] Nov 03 16:51:17.666724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:17.666623 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" podUID="8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" containerName="istio-proxy" containerID="cri-o://3fd441d421286b8c7998817076db3baea2526f99df4373c702b3c915fd87d411" gracePeriod=30 Nov 03 16:51:18.659440 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:18.659407 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7"] Nov 03 16:51:18.659902 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:18.659622 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" podUID="79da839f-f698-42eb-9275-9999cb9a1306" containerName="istio-proxy" containerID="cri-o://8b46d09daed868b7146f86e8676e14402d88717f73559001be08cc664448a784" gracePeriod=30 Nov 03 16:51:19.840688 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:19.840641 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:51:19.841119 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:19.840751 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" podUID="8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:51:21.341034 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:21.340999 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f"] Nov 03 16:51:21.341914 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:21.341862 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" podUID="e71ad661-ebdc-4bce-b870-a113baeba8e8" containerName="maas-api" containerID="cri-o://6eaa23c448af139b2f310860fa6ba9ca9323024d669d3e8c8db34045f1f4e5a6" gracePeriod=30 Nov 03 16:51:21.501087 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:21.501045 4947 patch_prober.go:29] interesting pod/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:51:21.501247 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:21.501106 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" podUID="8c8da48a-94af-4d09-aef7-21c1a69092c7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:51:22.088043 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.081939 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" Nov 03 16:51:22.115779 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.097432 4947 generic.go:356] "Generic (PLEG): container finished" podID="e71ad661-ebdc-4bce-b870-a113baeba8e8" containerID="6eaa23c448af139b2f310860fa6ba9ca9323024d669d3e8c8db34045f1f4e5a6" exitCode=0 Nov 03 16:51:22.115779 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.097606 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" event={"ID":"e71ad661-ebdc-4bce-b870-a113baeba8e8","Type":"ContainerDied","Data":"6eaa23c448af139b2f310860fa6ba9ca9323024d669d3e8c8db34045f1f4e5a6"} Nov 03 16:51:22.115779 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.097674 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" event={"ID":"e71ad661-ebdc-4bce-b870-a113baeba8e8","Type":"ContainerDied","Data":"1bdab06a8ee44dd63cd8d8a42af382f44b16b43ed3adf22f32c31d3253b6845c"} Nov 03 16:51:22.116889 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.116578 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f" Nov 03 16:51:22.117237 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.117063 4947 scope.go:122] "RemoveContainer" containerID="6eaa23c448af139b2f310860fa6ba9ca9323024d669d3e8c8db34045f1f4e5a6" Nov 03 16:51:22.169097 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.163347 4947 scope.go:122] "RemoveContainer" containerID="6eaa23c448af139b2f310860fa6ba9ca9323024d669d3e8c8db34045f1f4e5a6" Nov 03 16:51:22.169484 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:51:22.169380 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eaa23c448af139b2f310860fa6ba9ca9323024d669d3e8c8db34045f1f4e5a6\": container with ID starting with 6eaa23c448af139b2f310860fa6ba9ca9323024d669d3e8c8db34045f1f4e5a6 not found: ID does not exist" containerID="6eaa23c448af139b2f310860fa6ba9ca9323024d669d3e8c8db34045f1f4e5a6" Nov 03 16:51:22.169484 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.169424 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eaa23c448af139b2f310860fa6ba9ca9323024d669d3e8c8db34045f1f4e5a6"} err="failed to get container status \"6eaa23c448af139b2f310860fa6ba9ca9323024d669d3e8c8db34045f1f4e5a6\": rpc error: code = NotFound desc = could not find container \"6eaa23c448af139b2f310860fa6ba9ca9323024d669d3e8c8db34045f1f4e5a6\": container with ID starting with 6eaa23c448af139b2f310860fa6ba9ca9323024d669d3e8c8db34045f1f4e5a6 not found: ID does not exist" Nov 03 16:51:22.203513 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:51:22.203476 4947 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8da48a_94af_4d09_aef7_21c1a69092c7.slice/crio-4e7780711b3da942e5e1abad8f7b2e0a3bb094e22119043e2282a9398ed212ea.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8da48a_94af_4d09_aef7_21c1a69092c7.slice/crio-conmon-4e7780711b3da942e5e1abad8f7b2e0a3bb094e22119043e2282a9398ed212ea.scope\": RecentStats: unable to find data in memory cache]" Nov 03 16:51:22.203778 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:51:22.203745 4947 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8da48a_94af_4d09_aef7_21c1a69092c7.slice/crio-4e7780711b3da942e5e1abad8f7b2e0a3bb094e22119043e2282a9398ed212ea.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8da48a_94af_4d09_aef7_21c1a69092c7.slice/crio-conmon-4e7780711b3da942e5e1abad8f7b2e0a3bb094e22119043e2282a9398ed212ea.scope\": RecentStats: unable to find data in memory cache]" Nov 03 16:51:22.282548 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.282461 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e71ad661-ebdc-4bce-b870-a113baeba8e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e71ad661-ebdc-4bce-b870-a113baeba8e8-maas-api-tls\") pod \"e71ad661-ebdc-4bce-b870-a113baeba8e8\" (UID: \"e71ad661-ebdc-4bce-b870-a113baeba8e8\") " Nov 03 16:51:22.282768 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.282609 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e71ad661-ebdc-4bce-b870-a113baeba8e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71ad661-ebdc-4bce-b870-a113baeba8e8-openshift-service-ca\") pod \"e71ad661-ebdc-4bce-b870-a113baeba8e8\" (UID: \"e71ad661-ebdc-4bce-b870-a113baeba8e8\") " Nov 03 16:51:22.282867 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.282847 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71ad661-ebdc-4bce-b870-a113baeba8e8-kube-api-access-5d5fl\" (UniqueName: \"kubernetes.io/projected/e71ad661-ebdc-4bce-b870-a113baeba8e8-kube-api-access-5d5fl\") pod \"e71ad661-ebdc-4bce-b870-a113baeba8e8\" (UID: \"e71ad661-ebdc-4bce-b870-a113baeba8e8\") " Nov 03 16:51:22.283189 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.283162 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71ad661-ebdc-4bce-b870-a113baeba8e8-openshift-service-ca" pod "e71ad661-ebdc-4bce-b870-a113baeba8e8" (UID: "e71ad661-ebdc-4bce-b870-a113baeba8e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:22.283271 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.283259 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e71ad661-ebdc-4bce-b870-a113baeba8e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e71ad661-ebdc-4bce-b870-a113baeba8e8-metrics-certs\") pod \"e71ad661-ebdc-4bce-b870-a113baeba8e8\" (UID: \"e71ad661-ebdc-4bce-b870-a113baeba8e8\") " Nov 03 16:51:22.284050 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.284024 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e71ad661-ebdc-4bce-b870-a113baeba8e8-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.286552 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.286511 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71ad661-ebdc-4bce-b870-a113baeba8e8-kube-api-access-5d5fl" pod "e71ad661-ebdc-4bce-b870-a113baeba8e8" (UID: "e71ad661-ebdc-4bce-b870-a113baeba8e8"). InnerVolumeSpecName "kube-api-access-5d5fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:22.287643 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.287617 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e71ad661-ebdc-4bce-b870-a113baeba8e8-metrics-certs" pod "e71ad661-ebdc-4bce-b870-a113baeba8e8" (UID: "e71ad661-ebdc-4bce-b870-a113baeba8e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:51:22.291953 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.291929 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e71ad661-ebdc-4bce-b870-a113baeba8e8-maas-api-tls" pod "e71ad661-ebdc-4bce-b870-a113baeba8e8" (UID: "e71ad661-ebdc-4bce-b870-a113baeba8e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:51:22.385535 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.385502 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d5fl\" (UniqueName: \"kubernetes.io/projected/e71ad661-ebdc-4bce-b870-a113baeba8e8-kube-api-access-5d5fl\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.386023 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.385538 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e71ad661-ebdc-4bce-b870-a113baeba8e8-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.386023 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.385554 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e71ad661-ebdc-4bce-b870-a113baeba8e8-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.397660 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.397636 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:51:22.489081 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.489042 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f"] Nov 03 16:51:22.495741 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.494538 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d6bb42-5bff858b88-v4m9f"] Nov 03 16:51:22.598795 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.597188 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e71ad661-ebdc-4bce-b870-a113baeba8e8" path="/var/lib/kubelet/pods/e71ad661-ebdc-4bce-b870-a113baeba8e8/volumes" Nov 03 16:51:22.614722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.612460 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-data\") pod \"8c8da48a-94af-4d09-aef7-21c1a69092c7\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " Nov 03 16:51:22.614722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.612617 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-workload-certs\") pod \"8c8da48a-94af-4d09-aef7-21c1a69092c7\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " Nov 03 16:51:22.614722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.612653 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-workload-socket\") pod \"8c8da48a-94af-4d09-aef7-21c1a69092c7\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " Nov 03 16:51:22.614722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.612738 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-credential-socket\") pod \"8c8da48a-94af-4d09-aef7-21c1a69092c7\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " Nov 03 16:51:22.614722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.612818 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-envoy\") pod \"8c8da48a-94af-4d09-aef7-21c1a69092c7\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " Nov 03 16:51:22.614722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.612856 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-podinfo\") pod \"8c8da48a-94af-4d09-aef7-21c1a69092c7\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " Nov 03 16:51:22.614722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.612915 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-token\" (UniqueName: \"kubernetes.io/projected/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-token\") pod \"8c8da48a-94af-4d09-aef7-21c1a69092c7\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " Nov 03 16:51:22.614722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.612955 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8da48a-94af-4d09-aef7-21c1a69092c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8da48a-94af-4d09-aef7-21c1a69092c7-istiod-ca-cert\") pod \"8c8da48a-94af-4d09-aef7-21c1a69092c7\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " Nov 03 16:51:22.614722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.613023 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8da48a-94af-4d09-aef7-21c1a69092c7-kube-api-access-r7fst\" (UniqueName: \"kubernetes.io/projected/8c8da48a-94af-4d09-aef7-21c1a69092c7-kube-api-access-r7fst\") pod \"8c8da48a-94af-4d09-aef7-21c1a69092c7\" (UID: \"8c8da48a-94af-4d09-aef7-21c1a69092c7\") " Nov 03 16:51:22.629996 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.624144 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-credential-socket" pod "8c8da48a-94af-4d09-aef7-21c1a69092c7" (UID: "8c8da48a-94af-4d09-aef7-21c1a69092c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:22.629996 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.624357 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-data" pod "8c8da48a-94af-4d09-aef7-21c1a69092c7" (UID: "8c8da48a-94af-4d09-aef7-21c1a69092c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:22.629996 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.624530 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-workload-certs" pod "8c8da48a-94af-4d09-aef7-21c1a69092c7" (UID: "8c8da48a-94af-4d09-aef7-21c1a69092c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:22.629996 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.624749 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-workload-socket" pod "8c8da48a-94af-4d09-aef7-21c1a69092c7" (UID: "8c8da48a-94af-4d09-aef7-21c1a69092c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:22.629996 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.628258 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8da48a-94af-4d09-aef7-21c1a69092c7-istiod-ca-cert" pod "8c8da48a-94af-4d09-aef7-21c1a69092c7" (UID: "8c8da48a-94af-4d09-aef7-21c1a69092c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:22.632480 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.632446 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8da48a-94af-4d09-aef7-21c1a69092c7-kube-api-access-r7fst" pod "8c8da48a-94af-4d09-aef7-21c1a69092c7" (UID: "8c8da48a-94af-4d09-aef7-21c1a69092c7"). InnerVolumeSpecName "kube-api-access-r7fst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:22.633815 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.633777 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-podinfo" pod "8c8da48a-94af-4d09-aef7-21c1a69092c7" (UID: "8c8da48a-94af-4d09-aef7-21c1a69092c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:51:22.635131 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.635109 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-envoy" pod "8c8da48a-94af-4d09-aef7-21c1a69092c7" (UID: "8c8da48a-94af-4d09-aef7-21c1a69092c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:22.637441 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.637420 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-token" pod "8c8da48a-94af-4d09-aef7-21c1a69092c7" (UID: "8c8da48a-94af-4d09-aef7-21c1a69092c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:22.714940 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.714910 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.714940 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.714941 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.715125 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.714953 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.715125 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.714965 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.715125 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.714977 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.715125 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.714987 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.715125 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.714998 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c8da48a-94af-4d09-aef7-21c1a69092c7-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.715125 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.715009 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7fst\" (UniqueName: \"kubernetes.io/projected/8c8da48a-94af-4d09-aef7-21c1a69092c7-kube-api-access-r7fst\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.715125 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.715020 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c8da48a-94af-4d09-aef7-21c1a69092c7-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:22.930863 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:22.930841 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:51:23.106225 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.106137 4947 generic.go:356] "Generic (PLEG): container finished" podID="8c8da48a-94af-4d09-aef7-21c1a69092c7" containerID="4e7780711b3da942e5e1abad8f7b2e0a3bb094e22119043e2282a9398ed212ea" exitCode=0 Nov 03 16:51:23.106225 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.106183 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" Nov 03 16:51:23.106462 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.106223 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" event={"ID":"8c8da48a-94af-4d09-aef7-21c1a69092c7","Type":"ContainerDied","Data":"4e7780711b3da942e5e1abad8f7b2e0a3bb094e22119043e2282a9398ed212ea"} Nov 03 16:51:23.106462 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.106285 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5" event={"ID":"8c8da48a-94af-4d09-aef7-21c1a69092c7","Type":"ContainerDied","Data":"f18ccf2411168df660c9def6433d26d233dfe76bded1a8c505cb6fda160067d7"} Nov 03 16:51:23.106462 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.106308 4947 scope.go:122] "RemoveContainer" containerID="4e7780711b3da942e5e1abad8f7b2e0a3bb094e22119043e2282a9398ed212ea" Nov 03 16:51:23.108282 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.108260 4947 generic.go:356] "Generic (PLEG): container finished" podID="8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" containerID="3fd441d421286b8c7998817076db3baea2526f99df4373c702b3c915fd87d411" exitCode=0 Nov 03 16:51:23.108407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.108315 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" Nov 03 16:51:23.108407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.108343 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" event={"ID":"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20","Type":"ContainerDied","Data":"3fd441d421286b8c7998817076db3baea2526f99df4373c702b3c915fd87d411"} Nov 03 16:51:23.108407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.108389 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755" event={"ID":"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20","Type":"ContainerDied","Data":"85b3b993caf02578af3d8fb9994d5ddf34e6d70573feff0d72349db781f9b2de"} Nov 03 16:51:23.119278 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.119255 4947 scope.go:122] "RemoveContainer" containerID="4e7780711b3da942e5e1abad8f7b2e0a3bb094e22119043e2282a9398ed212ea" Nov 03 16:51:23.119788 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:51:23.119764 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e7780711b3da942e5e1abad8f7b2e0a3bb094e22119043e2282a9398ed212ea\": container with ID starting with 4e7780711b3da942e5e1abad8f7b2e0a3bb094e22119043e2282a9398ed212ea not found: ID does not exist" containerID="4e7780711b3da942e5e1abad8f7b2e0a3bb094e22119043e2282a9398ed212ea" Nov 03 16:51:23.119901 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.119795 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e7780711b3da942e5e1abad8f7b2e0a3bb094e22119043e2282a9398ed212ea"} err="failed to get container status \"4e7780711b3da942e5e1abad8f7b2e0a3bb094e22119043e2282a9398ed212ea\": rpc error: code = NotFound desc = could not find container \"4e7780711b3da942e5e1abad8f7b2e0a3bb094e22119043e2282a9398ed212ea\": container with ID starting with 4e7780711b3da942e5e1abad8f7b2e0a3bb094e22119043e2282a9398ed212ea not found: ID does not exist" Nov 03 16:51:23.119901 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.119814 4947 scope.go:122] "RemoveContainer" containerID="3fd441d421286b8c7998817076db3baea2526f99df4373c702b3c915fd87d411" Nov 03 16:51:23.120099 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.119958 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-credential-socket\") pod \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " Nov 03 16:51:23.120099 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.120042 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-podinfo\") pod \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " Nov 03 16:51:23.120099 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.120079 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-token\" (UniqueName: \"kubernetes.io/projected/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-token\") pod \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " Nov 03 16:51:23.120240 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.120123 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-workload-socket\") pod \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " Nov 03 16:51:23.120299 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.120279 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-credential-socket" pod "8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" (UID: "8390d6a9-fcf7-4ca1-afbe-4be83baf8c20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:23.120567 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.120545 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-workload-socket" pod "8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" (UID: "8390d6a9-fcf7-4ca1-afbe-4be83baf8c20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:23.120952 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.120932 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-kube-api-access-wbbtm\" (UniqueName: \"kubernetes.io/projected/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-kube-api-access-wbbtm\") pod \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " Nov 03 16:51:23.121085 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.121062 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-data\") pod \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " Nov 03 16:51:23.121141 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.121104 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-envoy\") pod \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " Nov 03 16:51:23.121186 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.121139 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istiod-ca-cert\") pod \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " Nov 03 16:51:23.121220 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.121192 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-workload-certs\") pod \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\" (UID: \"8390d6a9-fcf7-4ca1-afbe-4be83baf8c20\") " Nov 03 16:51:23.121342 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.121323 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-data" pod "8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" (UID: "8390d6a9-fcf7-4ca1-afbe-4be83baf8c20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:23.121601 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.121577 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istiod-ca-cert" pod "8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" (UID: "8390d6a9-fcf7-4ca1-afbe-4be83baf8c20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:23.121849 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.121828 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-workload-certs" pod "8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" (UID: "8390d6a9-fcf7-4ca1-afbe-4be83baf8c20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:23.121930 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.121900 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:23.121930 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.121918 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:23.122044 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.121930 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:23.122044 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.121943 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:23.123009 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.122979 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-podinfo" pod "8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" (UID: "8390d6a9-fcf7-4ca1-afbe-4be83baf8c20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:51:23.123114 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.123015 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-token" pod "8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" (UID: "8390d6a9-fcf7-4ca1-afbe-4be83baf8c20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:23.131380 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.131362 4947 scope.go:122] "RemoveContainer" containerID="1bd0a0d962da6a0aa8a1b0b5ddd7aea25febefee980fe34c58319351bc291dcb" Nov 03 16:51:23.132060 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.132039 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-kube-api-access-wbbtm" pod "8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" (UID: "8390d6a9-fcf7-4ca1-afbe-4be83baf8c20"). InnerVolumeSpecName "kube-api-access-wbbtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:23.140116 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.140093 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-envoy" pod "8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" (UID: "8390d6a9-fcf7-4ca1-afbe-4be83baf8c20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:23.140673 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.140657 4947 scope.go:122] "RemoveContainer" containerID="3fd441d421286b8c7998817076db3baea2526f99df4373c702b3c915fd87d411" Nov 03 16:51:23.140990 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:51:23.140970 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd441d421286b8c7998817076db3baea2526f99df4373c702b3c915fd87d411\": container with ID starting with 3fd441d421286b8c7998817076db3baea2526f99df4373c702b3c915fd87d411 not found: ID does not exist" containerID="3fd441d421286b8c7998817076db3baea2526f99df4373c702b3c915fd87d411" Nov 03 16:51:23.141044 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.140999 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd441d421286b8c7998817076db3baea2526f99df4373c702b3c915fd87d411"} err="failed to get container status \"3fd441d421286b8c7998817076db3baea2526f99df4373c702b3c915fd87d411\": rpc error: code = NotFound desc = could not find container \"3fd441d421286b8c7998817076db3baea2526f99df4373c702b3c915fd87d411\": container with ID starting with 3fd441d421286b8c7998817076db3baea2526f99df4373c702b3c915fd87d411 not found: ID does not exist" Nov 03 16:51:23.141044 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.141016 4947 scope.go:122] "RemoveContainer" containerID="1bd0a0d962da6a0aa8a1b0b5ddd7aea25febefee980fe34c58319351bc291dcb" Nov 03 16:51:23.141275 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:51:23.141259 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bd0a0d962da6a0aa8a1b0b5ddd7aea25febefee980fe34c58319351bc291dcb\": container with ID starting with 1bd0a0d962da6a0aa8a1b0b5ddd7aea25febefee980fe34c58319351bc291dcb not found: ID does not exist" containerID="1bd0a0d962da6a0aa8a1b0b5ddd7aea25febefee980fe34c58319351bc291dcb" Nov 03 16:51:23.141319 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.141285 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd0a0d962da6a0aa8a1b0b5ddd7aea25febefee980fe34c58319351bc291dcb"} err="failed to get container status \"1bd0a0d962da6a0aa8a1b0b5ddd7aea25febefee980fe34c58319351bc291dcb\": rpc error: code = NotFound desc = could not find container \"1bd0a0d962da6a0aa8a1b0b5ddd7aea25febefee980fe34c58319351bc291dcb\": container with ID starting with 1bd0a0d962da6a0aa8a1b0b5ddd7aea25febefee980fe34c58319351bc291dcb not found: ID does not exist" Nov 03 16:51:23.179488 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.179458 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5"] Nov 03 16:51:23.185307 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.185281 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-5svv5"] Nov 03 16:51:23.223451 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.223413 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:23.223451 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.223444 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:23.223666 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.223471 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:23.223666 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.223482 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbbtm\" (UniqueName: \"kubernetes.io/projected/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-kube-api-access-wbbtm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:23.223666 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.223496 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:23.459064 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.459036 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755"] Nov 03 16:51:23.463253 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.463219 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rg755"] Nov 03 16:51:23.940227 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:23.940153 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:51:24.119713 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.119668 4947 generic.go:356] "Generic (PLEG): container finished" podID="79da839f-f698-42eb-9275-9999cb9a1306" containerID="8b46d09daed868b7146f86e8676e14402d88717f73559001be08cc664448a784" exitCode=0 Nov 03 16:51:24.119889 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.119749 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" Nov 03 16:51:24.119889 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.119755 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" event={"ID":"79da839f-f698-42eb-9275-9999cb9a1306","Type":"ContainerDied","Data":"8b46d09daed868b7146f86e8676e14402d88717f73559001be08cc664448a784"} Nov 03 16:51:24.119889 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.119821 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7" event={"ID":"79da839f-f698-42eb-9275-9999cb9a1306","Type":"ContainerDied","Data":"1632baf4a5e9fabae6f8f561f7f57b7dd7c11a2c5fe39dd9eafbd02e744fb643"} Nov 03 16:51:24.119889 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.119849 4947 scope.go:122] "RemoveContainer" containerID="8b46d09daed868b7146f86e8676e14402d88717f73559001be08cc664448a784" Nov 03 16:51:24.130026 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.130000 4947 scope.go:122] "RemoveContainer" containerID="dfd75461b6ce7d634582b29fe24a6e8fa4f494a0780227ecd729038ccaa03a2f" Nov 03 16:51:24.133962 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.133936 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79da839f-f698-42eb-9275-9999cb9a1306-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79da839f-f698-42eb-9275-9999cb9a1306-istio-podinfo\") pod \"79da839f-f698-42eb-9275-9999cb9a1306\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " Nov 03 16:51:24.134081 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.134050 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79da839f-f698-42eb-9275-9999cb9a1306-istio-token\" (UniqueName: \"kubernetes.io/projected/79da839f-f698-42eb-9275-9999cb9a1306-istio-token\") pod \"79da839f-f698-42eb-9275-9999cb9a1306\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " Nov 03 16:51:24.134155 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.134092 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79da839f-f698-42eb-9275-9999cb9a1306-kube-api-access-cmm8r\" (UniqueName: \"kubernetes.io/projected/79da839f-f698-42eb-9275-9999cb9a1306-kube-api-access-cmm8r\") pod \"79da839f-f698-42eb-9275-9999cb9a1306\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " Nov 03 16:51:24.134155 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.134141 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-workload-socket\") pod \"79da839f-f698-42eb-9275-9999cb9a1306\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " Nov 03 16:51:24.134329 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.134302 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79da839f-f698-42eb-9275-9999cb9a1306-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79da839f-f698-42eb-9275-9999cb9a1306-istiod-ca-cert\") pod \"79da839f-f698-42eb-9275-9999cb9a1306\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " Nov 03 16:51:24.134407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.134386 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-workload-socket" pod "79da839f-f698-42eb-9275-9999cb9a1306" (UID: "79da839f-f698-42eb-9275-9999cb9a1306"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:24.134625 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.134605 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-credential-socket\") pod \"79da839f-f698-42eb-9275-9999cb9a1306\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " Nov 03 16:51:24.134742 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.134673 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-istio-data\") pod \"79da839f-f698-42eb-9275-9999cb9a1306\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " Nov 03 16:51:24.134803 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.134753 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79da839f-f698-42eb-9275-9999cb9a1306-istiod-ca-cert" pod "79da839f-f698-42eb-9275-9999cb9a1306" (UID: "79da839f-f698-42eb-9275-9999cb9a1306"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:51:24.134803 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.134762 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-workload-certs\") pod \"79da839f-f698-42eb-9275-9999cb9a1306\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " Nov 03 16:51:24.134910 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.134797 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-istio-envoy\") pod \"79da839f-f698-42eb-9275-9999cb9a1306\" (UID: \"79da839f-f698-42eb-9275-9999cb9a1306\") " Nov 03 16:51:24.134957 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.134920 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-credential-socket" pod "79da839f-f698-42eb-9275-9999cb9a1306" (UID: "79da839f-f698-42eb-9275-9999cb9a1306"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:24.135011 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.134986 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-istio-data" pod "79da839f-f698-42eb-9275-9999cb9a1306" (UID: "79da839f-f698-42eb-9275-9999cb9a1306"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:24.135500 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.135117 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-workload-certs" pod "79da839f-f698-42eb-9275-9999cb9a1306" (UID: "79da839f-f698-42eb-9275-9999cb9a1306"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:24.135500 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.135431 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:24.135500 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.135446 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:24.135500 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.135454 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79da839f-f698-42eb-9275-9999cb9a1306-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:24.135500 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.135462 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:24.135500 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.135472 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:24.137027 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.136995 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79da839f-f698-42eb-9275-9999cb9a1306-kube-api-access-cmm8r" pod "79da839f-f698-42eb-9275-9999cb9a1306" (UID: "79da839f-f698-42eb-9275-9999cb9a1306"). InnerVolumeSpecName "kube-api-access-cmm8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:24.137181 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.137159 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79da839f-f698-42eb-9275-9999cb9a1306-istio-token" pod "79da839f-f698-42eb-9275-9999cb9a1306" (UID: "79da839f-f698-42eb-9275-9999cb9a1306"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:51:24.137331 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.137316 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79da839f-f698-42eb-9275-9999cb9a1306-istio-podinfo" pod "79da839f-f698-42eb-9275-9999cb9a1306" (UID: "79da839f-f698-42eb-9275-9999cb9a1306"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:51:24.137432 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.137408 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-istio-envoy" pod "79da839f-f698-42eb-9275-9999cb9a1306" (UID: "79da839f-f698-42eb-9275-9999cb9a1306"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:51:24.142644 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.142626 4947 scope.go:122] "RemoveContainer" containerID="8b46d09daed868b7146f86e8676e14402d88717f73559001be08cc664448a784" Nov 03 16:51:24.142982 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:51:24.142964 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b46d09daed868b7146f86e8676e14402d88717f73559001be08cc664448a784\": container with ID starting with 8b46d09daed868b7146f86e8676e14402d88717f73559001be08cc664448a784 not found: ID does not exist" containerID="8b46d09daed868b7146f86e8676e14402d88717f73559001be08cc664448a784" Nov 03 16:51:24.143044 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.142990 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b46d09daed868b7146f86e8676e14402d88717f73559001be08cc664448a784"} err="failed to get container status \"8b46d09daed868b7146f86e8676e14402d88717f73559001be08cc664448a784\": rpc error: code = NotFound desc = could not find container \"8b46d09daed868b7146f86e8676e14402d88717f73559001be08cc664448a784\": container with ID starting with 8b46d09daed868b7146f86e8676e14402d88717f73559001be08cc664448a784 not found: ID does not exist" Nov 03 16:51:24.143044 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.143010 4947 scope.go:122] "RemoveContainer" containerID="dfd75461b6ce7d634582b29fe24a6e8fa4f494a0780227ecd729038ccaa03a2f" Nov 03 16:51:24.143296 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:51:24.143282 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfd75461b6ce7d634582b29fe24a6e8fa4f494a0780227ecd729038ccaa03a2f\": container with ID starting with dfd75461b6ce7d634582b29fe24a6e8fa4f494a0780227ecd729038ccaa03a2f not found: ID does not exist" containerID="dfd75461b6ce7d634582b29fe24a6e8fa4f494a0780227ecd729038ccaa03a2f" Nov 03 16:51:24.143346 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.143297 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd75461b6ce7d634582b29fe24a6e8fa4f494a0780227ecd729038ccaa03a2f"} err="failed to get container status \"dfd75461b6ce7d634582b29fe24a6e8fa4f494a0780227ecd729038ccaa03a2f\": rpc error: code = NotFound desc = could not find container \"dfd75461b6ce7d634582b29fe24a6e8fa4f494a0780227ecd729038ccaa03a2f\": container with ID starting with dfd75461b6ce7d634582b29fe24a6e8fa4f494a0780227ecd729038ccaa03a2f not found: ID does not exist" Nov 03 16:51:24.266299 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.255353 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79da839f-f698-42eb-9275-9999cb9a1306-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:24.266299 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.255405 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79da839f-f698-42eb-9275-9999cb9a1306-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:24.266299 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.255420 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79da839f-f698-42eb-9275-9999cb9a1306-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:24.266299 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.255504 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmm8r\" (UniqueName: \"kubernetes.io/projected/79da839f-f698-42eb-9275-9999cb9a1306-kube-api-access-cmm8r\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:51:24.471932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.471897 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7"] Nov 03 16:51:24.485545 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.485508 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-8gbj7"] Nov 03 16:51:24.566765 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.566649 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79da839f-f698-42eb-9275-9999cb9a1306" path="/var/lib/kubelet/pods/79da839f-f698-42eb-9275-9999cb9a1306/volumes" Nov 03 16:51:24.567184 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.567169 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" path="/var/lib/kubelet/pods/8390d6a9-fcf7-4ca1-afbe-4be83baf8c20/volumes" Nov 03 16:51:24.567586 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:24.567573 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c8da48a-94af-4d09-aef7-21c1a69092c7" path="/var/lib/kubelet/pods/8c8da48a-94af-4d09-aef7-21c1a69092c7/volumes" Nov 03 16:51:29.914996 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.914957 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z"] Nov 03 16:51:29.915893 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.915866 4947 cpu_manager.go:414] "containerMap: removing container" podUID="79da839f-f698-42eb-9275-9999cb9a1306" containerName="istio-proxy" Nov 03 16:51:29.915893 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.915893 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79da839f-f698-42eb-9275-9999cb9a1306" containerName="istio-proxy" Nov 03 16:51:29.916078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.915917 4947 cpu_manager.go:414] "containerMap: removing container" podUID="79da839f-f698-42eb-9275-9999cb9a1306" containerName="istio-proxy" Nov 03 16:51:29.916078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.915924 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79da839f-f698-42eb-9275-9999cb9a1306" containerName="istio-proxy" Nov 03 16:51:29.916078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.915934 4947 cpu_manager.go:414] "containerMap: removing container" podUID="e71ad661-ebdc-4bce-b870-a113baeba8e8" containerName="maas-api" Nov 03 16:51:29.916078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.915940 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e71ad661-ebdc-4bce-b870-a113baeba8e8" containerName="maas-api" Nov 03 16:51:29.916078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.915955 4947 cpu_manager.go:414] "containerMap: removing container" podUID="51572f38-36e8-475b-9171-31a55d83e984" containerName="payload-pre-processing" Nov 03 16:51:29.916078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.915962 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51572f38-36e8-475b-9171-31a55d83e984" containerName="payload-pre-processing" Nov 03 16:51:29.916078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.915977 4947 cpu_manager.go:414] "containerMap: removing container" podUID="462f73ae-b874-4e71-b21b-5793533767ac" containerName="payload-processing" Nov 03 16:51:29.916078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.915983 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="462f73ae-b874-4e71-b21b-5793533767ac" containerName="payload-processing" Nov 03 16:51:29.916078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.915993 4947 cpu_manager.go:414] "containerMap: removing container" podUID="8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" containerName="istio-proxy" Nov 03 16:51:29.916078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.915999 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" containerName="istio-proxy" Nov 03 16:51:29.916078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.916016 4947 cpu_manager.go:414] "containerMap: removing container" podUID="38262c13-420b-43af-bb5e-1cb3c62e44b0" containerName="istio-proxy" Nov 03 16:51:29.916078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.916023 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38262c13-420b-43af-bb5e-1cb3c62e44b0" containerName="istio-proxy" Nov 03 16:51:29.916078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.916044 4947 cpu_manager.go:414] "containerMap: removing container" podUID="8c8da48a-94af-4d09-aef7-21c1a69092c7" containerName="istio-proxy" Nov 03 16:51:29.916078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.916052 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8da48a-94af-4d09-aef7-21c1a69092c7" containerName="istio-proxy" Nov 03 16:51:29.916078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.916070 4947 cpu_manager.go:414] "containerMap: removing container" podUID="8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" containerName="istio-proxy" Nov 03 16:51:29.916078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.916077 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" containerName="istio-proxy" Nov 03 16:51:29.916827 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.916240 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="79da839f-f698-42eb-9275-9999cb9a1306" containerName="istio-proxy" Nov 03 16:51:29.916827 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.916255 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e71ad661-ebdc-4bce-b870-a113baeba8e8" containerName="maas-api" Nov 03 16:51:29.916827 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.916266 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="51572f38-36e8-475b-9171-31a55d83e984" containerName="payload-pre-processing" Nov 03 16:51:29.916827 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.916277 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="462f73ae-b874-4e71-b21b-5793533767ac" containerName="payload-processing" Nov 03 16:51:29.916827 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.916287 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" containerName="istio-proxy" Nov 03 16:51:29.916827 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.916296 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="38262c13-420b-43af-bb5e-1cb3c62e44b0" containerName="istio-proxy" Nov 03 16:51:29.916827 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.916306 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8da48a-94af-4d09-aef7-21c1a69092c7" containerName="istio-proxy" Nov 03 16:51:29.916827 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.916316 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="8390d6a9-fcf7-4ca1-afbe-4be83baf8c20" containerName="istio-proxy" Nov 03 16:51:29.939472 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.938966 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z" Nov 03 16:51:29.952378 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.950904 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z"] Nov 03 16:51:29.976723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.970352 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-549271-dockercfg-6jjlg\"" Nov 03 16:51:29.976723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:29.970492 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-549271\"" Nov 03 16:51:30.084875 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.084839 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzrbr\" (UniqueName: \"kubernetes.io/projected/f4a02173-6525-4899-918a-8087a3b5a45d-kube-api-access-fzrbr\") pod \"payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z\" (UID: \"f4a02173-6525-4899-918a-8087a3b5a45d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z" Nov 03 16:51:30.085201 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.085173 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a02173-6525-4899-918a-8087a3b5a45d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z\" (UID: \"f4a02173-6525-4899-918a-8087a3b5a45d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z" Nov 03 16:51:30.086382 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.086362 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26"] Nov 03 16:51:30.087106 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.087092 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="79da839f-f698-42eb-9275-9999cb9a1306" containerName="istio-proxy" Nov 03 16:51:30.090555 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.090542 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26" Nov 03 16:51:30.117402 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.117370 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26"] Nov 03 16:51:30.134733 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:51:30.134671 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:51:30.186135 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.186045 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a02173-6525-4899-918a-8087a3b5a45d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z\" (UID: \"f4a02173-6525-4899-918a-8087a3b5a45d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z" Nov 03 16:51:30.186135 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.186098 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzrbr\" (UniqueName: \"kubernetes.io/projected/f4a02173-6525-4899-918a-8087a3b5a45d-kube-api-access-fzrbr\") pod \"payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z\" (UID: \"f4a02173-6525-4899-918a-8087a3b5a45d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z" Nov 03 16:51:30.186389 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.186340 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2d985a-3b44-4af4-bb44-6b5faa064127-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26\" (UID: \"9d2d985a-3b44-4af4-bb44-6b5faa064127\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26" Nov 03 16:51:30.186449 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.186388 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zprxf\" (UniqueName: \"kubernetes.io/projected/9d2d985a-3b44-4af4-bb44-6b5faa064127-kube-api-access-zprxf\") pod \"payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26\" (UID: \"9d2d985a-3b44-4af4-bb44-6b5faa064127\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26" Nov 03 16:51:30.186770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.186751 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a02173-6525-4899-918a-8087a3b5a45d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z\" (UID: \"f4a02173-6525-4899-918a-8087a3b5a45d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z" Nov 03 16:51:30.211751 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.211686 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzrbr\" (UniqueName: \"kubernetes.io/projected/f4a02173-6525-4899-918a-8087a3b5a45d-kube-api-access-fzrbr\") pod \"payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z\" (UID: \"f4a02173-6525-4899-918a-8087a3b5a45d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z" Nov 03 16:51:30.288041 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.288006 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2d985a-3b44-4af4-bb44-6b5faa064127-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26\" (UID: \"9d2d985a-3b44-4af4-bb44-6b5faa064127\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26" Nov 03 16:51:30.288041 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.288048 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zprxf\" (UniqueName: \"kubernetes.io/projected/9d2d985a-3b44-4af4-bb44-6b5faa064127-kube-api-access-zprxf\") pod \"payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26\" (UID: \"9d2d985a-3b44-4af4-bb44-6b5faa064127\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26" Nov 03 16:51:30.288685 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.288666 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2d985a-3b44-4af4-bb44-6b5faa064127-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26\" (UID: \"9d2d985a-3b44-4af4-bb44-6b5faa064127\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26" Nov 03 16:51:30.306858 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.306825 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z" Nov 03 16:51:30.313014 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.312987 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zprxf\" (UniqueName: \"kubernetes.io/projected/9d2d985a-3b44-4af4-bb44-6b5faa064127-kube-api-access-zprxf\") pod \"payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26\" (UID: \"9d2d985a-3b44-4af4-bb44-6b5faa064127\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26" Nov 03 16:51:30.403754 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.403677 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26" Nov 03 16:51:30.616929 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:51:30.616895 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a02173_6525_4899_918a_8087a3b5a45d.slice/crio-bb8345271dc68296bf9197e7950cfdb23fc50bf40a42e86900d80df6ca99a890 WatchSource:0}: Error finding container bb8345271dc68296bf9197e7950cfdb23fc50bf40a42e86900d80df6ca99a890: Status 404 returned error can't find the container with id bb8345271dc68296bf9197e7950cfdb23fc50bf40a42e86900d80df6ca99a890 Nov 03 16:51:30.617125 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.617067 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z"] Nov 03 16:51:30.767208 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:30.766731 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26"] Nov 03 16:51:31.170507 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:31.170412 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z" event={"ID":"f4a02173-6525-4899-918a-8087a3b5a45d","Type":"ContainerStarted","Data":"91cf28c6446cfc1f7f4abdff2fe2e0fb46ee11bb732c42ff19f4ce2f85651ebe"} Nov 03 16:51:31.170507 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:31.170483 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z" Nov 03 16:51:31.171013 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:31.170510 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z" event={"ID":"f4a02173-6525-4899-918a-8087a3b5a45d","Type":"ContainerStarted","Data":"bb8345271dc68296bf9197e7950cfdb23fc50bf40a42e86900d80df6ca99a890"} Nov 03 16:51:31.172227 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:31.172201 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26" event={"ID":"9d2d985a-3b44-4af4-bb44-6b5faa064127","Type":"ContainerStarted","Data":"6645e982fc8cae00596b6d6922987bc29780ab0524e8d30c2471c15b40702d4e"} Nov 03 16:51:31.172368 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:31.172251 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26" Nov 03 16:51:31.172368 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:31.172271 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26" event={"ID":"9d2d985a-3b44-4af4-bb44-6b5faa064127","Type":"ContainerStarted","Data":"4fdc0eb272665ce81592925167b3f1be7882f5f06a9f61d509804159c055248d"} Nov 03 16:51:31.218530 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:31.218470 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z" podStartSLOduration=2.21845069 podStartE2EDuration="2.21845069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:51:31.215039097 +0000 UTC m=+2234.804039066" watchObservedRunningTime="2026-11-03 16:51:31.21845069 +0000 UTC m=+2234.807450658" Nov 03 16:51:31.263108 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:31.263050 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26" podStartSLOduration=1.263036855 podStartE2EDuration="1.263036855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:51:31.259991288 +0000 UTC m=+2234.848991258" watchObservedRunningTime="2026-11-03 16:51:31.263036855 +0000 UTC m=+2234.852036856" Nov 03 16:51:38.843329 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:38.843279 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:51:42.179790 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:42.179762 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z" Nov 03 16:51:42.180257 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:51:42.179832 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26" Nov 03 16:52:00.132833 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:52:00.132794 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:52:16.495778 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:16.495743 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg"] Nov 03 16:52:16.496294 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:16.495963 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg" podUID="ff88d741-c433-40a3-804b-24742f4a8140" containerName="payload-processing" containerID="cri-o://1cd6fcdc81b7f31ddd2bc09a43fc7e388f87b3c262da66360c1b756059357b1c" gracePeriod=30 Nov 03 16:52:16.550687 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:16.550647 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn"] Nov 03 16:52:16.550933 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:16.550904 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn" podUID="007c9802-d9bf-4cd2-a062-6da9b70c31ff" containerName="payload-pre-processing" containerID="cri-o://945760414b599febdff619517e8718ca91ae1eec147b4dce751b41e6f88d1ce0" gracePeriod=30 Nov 03 16:52:16.633612 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:16.633582 4947 generic.go:356] "Generic (PLEG): container finished" podID="ff88d741-c433-40a3-804b-24742f4a8140" containerID="1cd6fcdc81b7f31ddd2bc09a43fc7e388f87b3c262da66360c1b756059357b1c" exitCode=0 Nov 03 16:52:16.633824 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:16.633649 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg" event={"ID":"ff88d741-c433-40a3-804b-24742f4a8140","Type":"ContainerDied","Data":"1cd6fcdc81b7f31ddd2bc09a43fc7e388f87b3c262da66360c1b756059357b1c"} Nov 03 16:52:16.796775 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:16.796731 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9"] Nov 03 16:52:16.797302 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:16.797224 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" podUID="a0adbe8a-947c-433a-886e-c824cc5d0566" containerName="maas-api" containerID="cri-o://5c3479ed84629b7471423c5f4626cfea4ad1186c46436dbbc5faa3d084a10024" gracePeriod=30 Nov 03 16:52:16.797654 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:16.797383 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg" Nov 03 16:52:16.914891 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:16.914856 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff88d741-c433-40a3-804b-24742f4a8140-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff88d741-c433-40a3-804b-24742f4a8140-plugins-config-volume\") pod \"ff88d741-c433-40a3-804b-24742f4a8140\" (UID: \"ff88d741-c433-40a3-804b-24742f4a8140\") " Nov 03 16:52:16.915070 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:16.915055 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff88d741-c433-40a3-804b-24742f4a8140-kube-api-access-dz8xz\" (UniqueName: \"kubernetes.io/projected/ff88d741-c433-40a3-804b-24742f4a8140-kube-api-access-dz8xz\") pod \"ff88d741-c433-40a3-804b-24742f4a8140\" (UID: \"ff88d741-c433-40a3-804b-24742f4a8140\") " Nov 03 16:52:16.916421 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:16.916389 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff88d741-c433-40a3-804b-24742f4a8140-plugins-config-volume" pod "ff88d741-c433-40a3-804b-24742f4a8140" (UID: "ff88d741-c433-40a3-804b-24742f4a8140"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:16.921413 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:16.921206 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff88d741-c433-40a3-804b-24742f4a8140-kube-api-access-dz8xz" pod "ff88d741-c433-40a3-804b-24742f4a8140" (UID: "ff88d741-c433-40a3-804b-24742f4a8140"). InnerVolumeSpecName "kube-api-access-dz8xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:16.923098 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:16.923073 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn" Nov 03 16:52:17.016464 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.016323 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007c9802-d9bf-4cd2-a062-6da9b70c31ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007c9802-d9bf-4cd2-a062-6da9b70c31ff-plugins-config-volume\") pod \"007c9802-d9bf-4cd2-a062-6da9b70c31ff\" (UID: \"007c9802-d9bf-4cd2-a062-6da9b70c31ff\") " Nov 03 16:52:17.017934 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.016536 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007c9802-d9bf-4cd2-a062-6da9b70c31ff-kube-api-access-h5njj\" (UniqueName: \"kubernetes.io/projected/007c9802-d9bf-4cd2-a062-6da9b70c31ff-kube-api-access-h5njj\") pod \"007c9802-d9bf-4cd2-a062-6da9b70c31ff\" (UID: \"007c9802-d9bf-4cd2-a062-6da9b70c31ff\") " Nov 03 16:52:17.017934 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.017261 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff88d741-c433-40a3-804b-24742f4a8140-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:17.017934 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.017286 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz8xz\" (UniqueName: \"kubernetes.io/projected/ff88d741-c433-40a3-804b-24742f4a8140-kube-api-access-dz8xz\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:17.017934 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.017487 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007c9802-d9bf-4cd2-a062-6da9b70c31ff-plugins-config-volume" pod "007c9802-d9bf-4cd2-a062-6da9b70c31ff" (UID: "007c9802-d9bf-4cd2-a062-6da9b70c31ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:17.020528 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.020497 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007c9802-d9bf-4cd2-a062-6da9b70c31ff-kube-api-access-h5njj" pod "007c9802-d9bf-4cd2-a062-6da9b70c31ff" (UID: "007c9802-d9bf-4cd2-a062-6da9b70c31ff"). InnerVolumeSpecName "kube-api-access-h5njj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:17.124737 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.121765 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/007c9802-d9bf-4cd2-a062-6da9b70c31ff-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:17.124737 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.121804 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5njj\" (UniqueName: \"kubernetes.io/projected/007c9802-d9bf-4cd2-a062-6da9b70c31ff-kube-api-access-h5njj\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:17.383759 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.382410 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:52:17.457521 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.457483 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0adbe8a-947c-433a-886e-c824cc5d0566-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0adbe8a-947c-433a-886e-c824cc5d0566-openshift-service-ca\") pod \"a0adbe8a-947c-433a-886e-c824cc5d0566\" (UID: \"a0adbe8a-947c-433a-886e-c824cc5d0566\") " Nov 03 16:52:17.457750 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.457528 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0adbe8a-947c-433a-886e-c824cc5d0566-kube-api-access-5pvpl\" (UniqueName: \"kubernetes.io/projected/a0adbe8a-947c-433a-886e-c824cc5d0566-kube-api-access-5pvpl\") pod \"a0adbe8a-947c-433a-886e-c824cc5d0566\" (UID: \"a0adbe8a-947c-433a-886e-c824cc5d0566\") " Nov 03 16:52:17.457750 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.457548 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0adbe8a-947c-433a-886e-c824cc5d0566-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0adbe8a-947c-433a-886e-c824cc5d0566-metrics-certs\") pod \"a0adbe8a-947c-433a-886e-c824cc5d0566\" (UID: \"a0adbe8a-947c-433a-886e-c824cc5d0566\") " Nov 03 16:52:17.457750 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.457713 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0adbe8a-947c-433a-886e-c824cc5d0566-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0adbe8a-947c-433a-886e-c824cc5d0566-maas-api-tls\") pod \"a0adbe8a-947c-433a-886e-c824cc5d0566\" (UID: \"a0adbe8a-947c-433a-886e-c824cc5d0566\") " Nov 03 16:52:17.458033 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.457997 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0adbe8a-947c-433a-886e-c824cc5d0566-openshift-service-ca" pod "a0adbe8a-947c-433a-886e-c824cc5d0566" (UID: "a0adbe8a-947c-433a-886e-c824cc5d0566"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:17.458773 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.458749 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0adbe8a-947c-433a-886e-c824cc5d0566-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:17.460024 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.459984 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0adbe8a-947c-433a-886e-c824cc5d0566-metrics-certs" pod "a0adbe8a-947c-433a-886e-c824cc5d0566" (UID: "a0adbe8a-947c-433a-886e-c824cc5d0566"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:17.460128 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.460078 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0adbe8a-947c-433a-886e-c824cc5d0566-kube-api-access-5pvpl" pod "a0adbe8a-947c-433a-886e-c824cc5d0566" (UID: "a0adbe8a-947c-433a-886e-c824cc5d0566"). InnerVolumeSpecName "kube-api-access-5pvpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:17.460380 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.460366 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0adbe8a-947c-433a-886e-c824cc5d0566-maas-api-tls" pod "a0adbe8a-947c-433a-886e-c824cc5d0566" (UID: "a0adbe8a-947c-433a-886e-c824cc5d0566"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:17.559372 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.559343 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pvpl\" (UniqueName: \"kubernetes.io/projected/a0adbe8a-947c-433a-886e-c824cc5d0566-kube-api-access-5pvpl\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:17.559372 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.559370 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0adbe8a-947c-433a-886e-c824cc5d0566-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:17.559372 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.559379 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0adbe8a-947c-433a-886e-c824cc5d0566-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:17.641285 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.641200 4947 generic.go:356] "Generic (PLEG): container finished" podID="007c9802-d9bf-4cd2-a062-6da9b70c31ff" containerID="945760414b599febdff619517e8718ca91ae1eec147b4dce751b41e6f88d1ce0" exitCode=0 Nov 03 16:52:17.641285 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.641247 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn" Nov 03 16:52:17.641520 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.641281 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn" event={"ID":"007c9802-d9bf-4cd2-a062-6da9b70c31ff","Type":"ContainerDied","Data":"945760414b599febdff619517e8718ca91ae1eec147b4dce751b41e6f88d1ce0"} Nov 03 16:52:17.641520 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.641348 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn" event={"ID":"007c9802-d9bf-4cd2-a062-6da9b70c31ff","Type":"ContainerDied","Data":"62ba83f9fe7e122436b64fb9ccefb9d05893b32cce0782b1320cf627abde668c"} Nov 03 16:52:17.641520 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.641377 4947 scope.go:122] "RemoveContainer" containerID="945760414b599febdff619517e8718ca91ae1eec147b4dce751b41e6f88d1ce0" Nov 03 16:52:17.642664 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.642640 4947 generic.go:356] "Generic (PLEG): container finished" podID="a0adbe8a-947c-433a-886e-c824cc5d0566" containerID="5c3479ed84629b7471423c5f4626cfea4ad1186c46436dbbc5faa3d084a10024" exitCode=0 Nov 03 16:52:17.642824 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.642673 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" event={"ID":"a0adbe8a-947c-433a-886e-c824cc5d0566","Type":"ContainerDied","Data":"5c3479ed84629b7471423c5f4626cfea4ad1186c46436dbbc5faa3d084a10024"} Nov 03 16:52:17.642824 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.642729 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" Nov 03 16:52:17.642824 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.642741 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9" event={"ID":"a0adbe8a-947c-433a-886e-c824cc5d0566","Type":"ContainerDied","Data":"66f602b5efc82b741089ae8e73153f26fdf54d125b7d15461bb9858a7d34c5a1"} Nov 03 16:52:17.644285 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.644256 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg" event={"ID":"ff88d741-c433-40a3-804b-24742f4a8140","Type":"ContainerDied","Data":"910cf57f8995852d1af60c32e4a82f17798469327202a7d5f2817e703af9538f"} Nov 03 16:52:17.644382 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.644263 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg" Nov 03 16:52:17.653462 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.653439 4947 scope.go:122] "RemoveContainer" containerID="945760414b599febdff619517e8718ca91ae1eec147b4dce751b41e6f88d1ce0" Nov 03 16:52:17.653814 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:52:17.653778 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945760414b599febdff619517e8718ca91ae1eec147b4dce751b41e6f88d1ce0\": container with ID starting with 945760414b599febdff619517e8718ca91ae1eec147b4dce751b41e6f88d1ce0 not found: ID does not exist" containerID="945760414b599febdff619517e8718ca91ae1eec147b4dce751b41e6f88d1ce0" Nov 03 16:52:17.653908 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.653812 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945760414b599febdff619517e8718ca91ae1eec147b4dce751b41e6f88d1ce0"} err="failed to get container status \"945760414b599febdff619517e8718ca91ae1eec147b4dce751b41e6f88d1ce0\": rpc error: code = NotFound desc = could not find container \"945760414b599febdff619517e8718ca91ae1eec147b4dce751b41e6f88d1ce0\": container with ID starting with 945760414b599febdff619517e8718ca91ae1eec147b4dce751b41e6f88d1ce0 not found: ID does not exist" Nov 03 16:52:17.653908 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.653836 4947 scope.go:122] "RemoveContainer" containerID="5c3479ed84629b7471423c5f4626cfea4ad1186c46436dbbc5faa3d084a10024" Nov 03 16:52:17.673412 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.673390 4947 scope.go:122] "RemoveContainer" containerID="5c3479ed84629b7471423c5f4626cfea4ad1186c46436dbbc5faa3d084a10024" Nov 03 16:52:17.673732 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:52:17.673688 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c3479ed84629b7471423c5f4626cfea4ad1186c46436dbbc5faa3d084a10024\": container with ID starting with 5c3479ed84629b7471423c5f4626cfea4ad1186c46436dbbc5faa3d084a10024 not found: ID does not exist" containerID="5c3479ed84629b7471423c5f4626cfea4ad1186c46436dbbc5faa3d084a10024" Nov 03 16:52:17.673918 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.673740 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c3479ed84629b7471423c5f4626cfea4ad1186c46436dbbc5faa3d084a10024"} err="failed to get container status \"5c3479ed84629b7471423c5f4626cfea4ad1186c46436dbbc5faa3d084a10024\": rpc error: code = NotFound desc = could not find container \"5c3479ed84629b7471423c5f4626cfea4ad1186c46436dbbc5faa3d084a10024\": container with ID starting with 5c3479ed84629b7471423c5f4626cfea4ad1186c46436dbbc5faa3d084a10024 not found: ID does not exist" Nov 03 16:52:17.673918 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.673762 4947 scope.go:122] "RemoveContainer" containerID="1cd6fcdc81b7f31ddd2bc09a43fc7e388f87b3c262da66360c1b756059357b1c" Nov 03 16:52:17.702589 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.702568 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn"] Nov 03 16:52:17.705344 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.705325 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5a7d95-747644d846-lnvkn"] Nov 03 16:52:17.788803 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.788773 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg"] Nov 03 16:52:17.792155 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.792128 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5a7d95-6b8569f876-kd6cg"] Nov 03 16:52:17.823215 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.823186 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9"] Nov 03 16:52:17.829836 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:17.829815 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-223659-854f4f8d6c-c5gp9"] Nov 03 16:52:18.566367 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:18.566334 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="007c9802-d9bf-4cd2-a062-6da9b70c31ff" path="/var/lib/kubelet/pods/007c9802-d9bf-4cd2-a062-6da9b70c31ff/volumes" Nov 03 16:52:18.566847 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:18.566816 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0adbe8a-947c-433a-886e-c824cc5d0566" path="/var/lib/kubelet/pods/a0adbe8a-947c-433a-886e-c824cc5d0566/volumes" Nov 03 16:52:18.567309 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:18.567289 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff88d741-c433-40a3-804b-24742f4a8140" path="/var/lib/kubelet/pods/ff88d741-c433-40a3-804b-24742f4a8140/volumes" Nov 03 16:52:20.101556 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:20.101523 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k"] Nov 03 16:52:20.102015 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:20.101817 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" podUID="a707604b-dc82-4c5b-bdc4-9f88027df05c" containerName="istio-proxy" containerID="cri-o://92476e59fd61af2b994259bc879bf173121d6beca551490ae843a3aa9f373e41" gracePeriod=30 Nov 03 16:52:20.319193 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:20.319154 4947 patch_prober.go:29] interesting pod/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:52:20.319473 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:20.319446 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" podUID="a707604b-dc82-4c5b-bdc4-9f88027df05c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:52:25.386425 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.386400 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:52:25.427163 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.427132 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-credential-socket\") pod \"a707604b-dc82-4c5b-bdc4-9f88027df05c\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " Nov 03 16:52:25.427328 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.427186 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-podinfo\") pod \"a707604b-dc82-4c5b-bdc4-9f88027df05c\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " Nov 03 16:52:25.427328 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.427208 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-envoy\") pod \"a707604b-dc82-4c5b-bdc4-9f88027df05c\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " Nov 03 16:52:25.427328 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.427260 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-token\" (UniqueName: \"kubernetes.io/projected/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-token\") pod \"a707604b-dc82-4c5b-bdc4-9f88027df05c\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " Nov 03 16:52:25.427328 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.427281 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a707604b-dc82-4c5b-bdc4-9f88027df05c-kube-api-access-8npbh\" (UniqueName: \"kubernetes.io/projected/a707604b-dc82-4c5b-bdc4-9f88027df05c-kube-api-access-8npbh\") pod \"a707604b-dc82-4c5b-bdc4-9f88027df05c\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " Nov 03 16:52:25.427490 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.427434 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-credential-socket" pod "a707604b-dc82-4c5b-bdc4-9f88027df05c" (UID: "a707604b-dc82-4c5b-bdc4-9f88027df05c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:25.427490 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.427452 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a707604b-dc82-4c5b-bdc4-9f88027df05c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a707604b-dc82-4c5b-bdc4-9f88027df05c-istiod-ca-cert\") pod \"a707604b-dc82-4c5b-bdc4-9f88027df05c\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " Nov 03 16:52:25.427569 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.427498 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-workload-certs\") pod \"a707604b-dc82-4c5b-bdc4-9f88027df05c\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " Nov 03 16:52:25.427569 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.427558 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-data\") pod \"a707604b-dc82-4c5b-bdc4-9f88027df05c\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " Nov 03 16:52:25.427674 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.427621 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-workload-socket\") pod \"a707604b-dc82-4c5b-bdc4-9f88027df05c\" (UID: \"a707604b-dc82-4c5b-bdc4-9f88027df05c\") " Nov 03 16:52:25.427845 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.427822 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a707604b-dc82-4c5b-bdc4-9f88027df05c-istiod-ca-cert" pod "a707604b-dc82-4c5b-bdc4-9f88027df05c" (UID: "a707604b-dc82-4c5b-bdc4-9f88027df05c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:25.427986 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.427928 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-data" pod "a707604b-dc82-4c5b-bdc4-9f88027df05c" (UID: "a707604b-dc82-4c5b-bdc4-9f88027df05c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:25.428199 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.428175 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-workload-socket" pod "a707604b-dc82-4c5b-bdc4-9f88027df05c" (UID: "a707604b-dc82-4c5b-bdc4-9f88027df05c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:25.428268 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.428195 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-workload-certs" pod "a707604b-dc82-4c5b-bdc4-9f88027df05c" (UID: "a707604b-dc82-4c5b-bdc4-9f88027df05c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:25.428555 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.428535 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a707604b-dc82-4c5b-bdc4-9f88027df05c-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:25.428600 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.428563 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:25.428600 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.428579 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:25.428600 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.428593 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:25.428688 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.428605 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:25.429783 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.429758 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-token" pod "a707604b-dc82-4c5b-bdc4-9f88027df05c" (UID: "a707604b-dc82-4c5b-bdc4-9f88027df05c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:25.429850 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.429806 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a707604b-dc82-4c5b-bdc4-9f88027df05c-kube-api-access-8npbh" pod "a707604b-dc82-4c5b-bdc4-9f88027df05c" (UID: "a707604b-dc82-4c5b-bdc4-9f88027df05c"). InnerVolumeSpecName "kube-api-access-8npbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:25.430065 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.430045 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-podinfo" pod "a707604b-dc82-4c5b-bdc4-9f88027df05c" (UID: "a707604b-dc82-4c5b-bdc4-9f88027df05c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:52:25.430065 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.430055 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-envoy" pod "a707604b-dc82-4c5b-bdc4-9f88027df05c" (UID: "a707604b-dc82-4c5b-bdc4-9f88027df05c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:25.541492 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.539683 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:25.541492 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.541378 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8npbh\" (UniqueName: \"kubernetes.io/projected/a707604b-dc82-4c5b-bdc4-9f88027df05c-kube-api-access-8npbh\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:25.541492 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.541426 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:25.541492 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.541444 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a707604b-dc82-4c5b-bdc4-9f88027df05c-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:25.720376 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.720294 4947 generic.go:356] "Generic (PLEG): container finished" podID="a707604b-dc82-4c5b-bdc4-9f88027df05c" containerID="92476e59fd61af2b994259bc879bf173121d6beca551490ae843a3aa9f373e41" exitCode=0 Nov 03 16:52:25.720376 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.720343 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" Nov 03 16:52:25.720376 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.720357 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" event={"ID":"a707604b-dc82-4c5b-bdc4-9f88027df05c","Type":"ContainerDied","Data":"92476e59fd61af2b994259bc879bf173121d6beca551490ae843a3aa9f373e41"} Nov 03 16:52:25.720621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.720399 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k" event={"ID":"a707604b-dc82-4c5b-bdc4-9f88027df05c","Type":"ContainerDied","Data":"622176f0200dc840994a1a2cbf5a4764e8f224d1cd150affd6b611cd712bd1ce"} Nov 03 16:52:25.720621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.720419 4947 scope.go:122] "RemoveContainer" containerID="92476e59fd61af2b994259bc879bf173121d6beca551490ae843a3aa9f373e41" Nov 03 16:52:25.731076 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.731057 4947 scope.go:122] "RemoveContainer" containerID="7c27267fb1dce452385c60cb3154e80321b181d26c6d7b7e80f917d0ff8df9d5" Nov 03 16:52:25.740280 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.740263 4947 scope.go:122] "RemoveContainer" containerID="92476e59fd61af2b994259bc879bf173121d6beca551490ae843a3aa9f373e41" Nov 03 16:52:25.740542 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:52:25.740521 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92476e59fd61af2b994259bc879bf173121d6beca551490ae843a3aa9f373e41\": container with ID starting with 92476e59fd61af2b994259bc879bf173121d6beca551490ae843a3aa9f373e41 not found: ID does not exist" containerID="92476e59fd61af2b994259bc879bf173121d6beca551490ae843a3aa9f373e41" Nov 03 16:52:25.740613 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.740550 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92476e59fd61af2b994259bc879bf173121d6beca551490ae843a3aa9f373e41"} err="failed to get container status \"92476e59fd61af2b994259bc879bf173121d6beca551490ae843a3aa9f373e41\": rpc error: code = NotFound desc = could not find container \"92476e59fd61af2b994259bc879bf173121d6beca551490ae843a3aa9f373e41\": container with ID starting with 92476e59fd61af2b994259bc879bf173121d6beca551490ae843a3aa9f373e41 not found: ID does not exist" Nov 03 16:52:25.740613 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.740568 4947 scope.go:122] "RemoveContainer" containerID="7c27267fb1dce452385c60cb3154e80321b181d26c6d7b7e80f917d0ff8df9d5" Nov 03 16:52:25.740837 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:52:25.740820 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c27267fb1dce452385c60cb3154e80321b181d26c6d7b7e80f917d0ff8df9d5\": container with ID starting with 7c27267fb1dce452385c60cb3154e80321b181d26c6d7b7e80f917d0ff8df9d5 not found: ID does not exist" containerID="7c27267fb1dce452385c60cb3154e80321b181d26c6d7b7e80f917d0ff8df9d5" Nov 03 16:52:25.740883 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.740844 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c27267fb1dce452385c60cb3154e80321b181d26c6d7b7e80f917d0ff8df9d5"} err="failed to get container status \"7c27267fb1dce452385c60cb3154e80321b181d26c6d7b7e80f917d0ff8df9d5\": rpc error: code = NotFound desc = could not find container \"7c27267fb1dce452385c60cb3154e80321b181d26c6d7b7e80f917d0ff8df9d5\": container with ID starting with 7c27267fb1dce452385c60cb3154e80321b181d26c6d7b7e80f917d0ff8df9d5 not found: ID does not exist" Nov 03 16:52:25.779812 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.779773 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k"] Nov 03 16:52:25.792849 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:25.792820 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-5a7d95-openshift-default-b9d77d97c-x6l9k"] Nov 03 16:52:26.566129 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:26.566092 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a707604b-dc82-4c5b-bdc4-9f88027df05c" path="/var/lib/kubelet/pods/a707604b-dc82-4c5b-bdc4-9f88027df05c/volumes" Nov 03 16:52:30.085644 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:30.085544 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:52:30.131176 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:52:30.131143 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:52:31.664245 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:31.664213 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g"] Nov 03 16:52:34.839803 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:34.839772 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq"] Nov 03 16:52:34.840267 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:34.839982 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" podUID="3275cc93-2a98-42d2-9f4a-84ce9c10fbe5" containerName="maas-api" containerID="cri-o://8727ebbe07cce33501979af62f9149a7476e3751d3bbc469750335af2ae7f92a" gracePeriod=30 Nov 03 16:52:35.010574 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.010539 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26"] Nov 03 16:52:35.010875 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.010837 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26" podUID="9d2d985a-3b44-4af4-bb44-6b5faa064127" containerName="payload-processing" containerID="cri-o://6645e982fc8cae00596b6d6922987bc29780ab0524e8d30c2471c15b40702d4e" gracePeriod=30 Nov 03 16:52:35.033733 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.033663 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z"] Nov 03 16:52:35.033960 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.033915 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z" podUID="f4a02173-6525-4899-918a-8087a3b5a45d" containerName="payload-pre-processing" containerID="cri-o://91cf28c6446cfc1f7f4abdff2fe2e0fb46ee11bb732c42ff19f4ce2f85651ebe" gracePeriod=30 Nov 03 16:52:35.187782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.187754 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" Nov 03 16:52:35.305852 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.305824 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-metrics-certs\") pod \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\" (UID: \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\") " Nov 03 16:52:35.305993 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.305957 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-kube-api-access-fdc8h\" (UniqueName: \"kubernetes.io/projected/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-kube-api-access-fdc8h\") pod \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\" (UID: \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\") " Nov 03 16:52:35.305993 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.305988 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-maas-api-tls\") pod \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\" (UID: \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\") " Nov 03 16:52:35.306126 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.306106 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-openshift-service-ca\") pod \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\" (UID: \"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5\") " Nov 03 16:52:35.306765 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.306744 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-openshift-service-ca" pod "3275cc93-2a98-42d2-9f4a-84ce9c10fbe5" (UID: "3275cc93-2a98-42d2-9f4a-84ce9c10fbe5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:35.309438 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.309402 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-metrics-certs" pod "3275cc93-2a98-42d2-9f4a-84ce9c10fbe5" (UID: "3275cc93-2a98-42d2-9f4a-84ce9c10fbe5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:35.309742 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.309686 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-kube-api-access-fdc8h" pod "3275cc93-2a98-42d2-9f4a-84ce9c10fbe5" (UID: "3275cc93-2a98-42d2-9f4a-84ce9c10fbe5"). InnerVolumeSpecName "kube-api-access-fdc8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:35.310054 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.310029 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-maas-api-tls" pod "3275cc93-2a98-42d2-9f4a-84ce9c10fbe5" (UID: "3275cc93-2a98-42d2-9f4a-84ce9c10fbe5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:35.326578 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.326434 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26" Nov 03 16:52:35.437782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.430893 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:35.437782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.430935 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:35.437782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.430950 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:35.437782 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.430963 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdc8h\" (UniqueName: \"kubernetes.io/projected/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5-kube-api-access-fdc8h\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:35.537344 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.537296 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d2d985a-3b44-4af4-bb44-6b5faa064127-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2d985a-3b44-4af4-bb44-6b5faa064127-plugins-config-volume\") pod \"9d2d985a-3b44-4af4-bb44-6b5faa064127\" (UID: \"9d2d985a-3b44-4af4-bb44-6b5faa064127\") " Nov 03 16:52:35.537608 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.537588 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d2d985a-3b44-4af4-bb44-6b5faa064127-kube-api-access-zprxf\" (UniqueName: \"kubernetes.io/projected/9d2d985a-3b44-4af4-bb44-6b5faa064127-kube-api-access-zprxf\") pod \"9d2d985a-3b44-4af4-bb44-6b5faa064127\" (UID: \"9d2d985a-3b44-4af4-bb44-6b5faa064127\") " Nov 03 16:52:35.539887 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.538326 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d2d985a-3b44-4af4-bb44-6b5faa064127-plugins-config-volume" pod "9d2d985a-3b44-4af4-bb44-6b5faa064127" (UID: "9d2d985a-3b44-4af4-bb44-6b5faa064127"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:35.540867 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.540845 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d2d985a-3b44-4af4-bb44-6b5faa064127-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:35.542859 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.542111 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2d985a-3b44-4af4-bb44-6b5faa064127-kube-api-access-zprxf" pod "9d2d985a-3b44-4af4-bb44-6b5faa064127" (UID: "9d2d985a-3b44-4af4-bb44-6b5faa064127"). InnerVolumeSpecName "kube-api-access-zprxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:35.551041 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.551019 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z" Nov 03 16:52:35.642139 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.642108 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a02173-6525-4899-918a-8087a3b5a45d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a02173-6525-4899-918a-8087a3b5a45d-plugins-config-volume\") pod \"f4a02173-6525-4899-918a-8087a3b5a45d\" (UID: \"f4a02173-6525-4899-918a-8087a3b5a45d\") " Nov 03 16:52:35.642320 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.642310 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a02173-6525-4899-918a-8087a3b5a45d-kube-api-access-fzrbr\" (UniqueName: \"kubernetes.io/projected/f4a02173-6525-4899-918a-8087a3b5a45d-kube-api-access-fzrbr\") pod \"f4a02173-6525-4899-918a-8087a3b5a45d\" (UID: \"f4a02173-6525-4899-918a-8087a3b5a45d\") " Nov 03 16:52:35.642503 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.642480 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a02173-6525-4899-918a-8087a3b5a45d-plugins-config-volume" pod "f4a02173-6525-4899-918a-8087a3b5a45d" (UID: "f4a02173-6525-4899-918a-8087a3b5a45d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:35.642822 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.642798 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zprxf\" (UniqueName: \"kubernetes.io/projected/9d2d985a-3b44-4af4-bb44-6b5faa064127-kube-api-access-zprxf\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:35.642822 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.642826 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4a02173-6525-4899-918a-8087a3b5a45d-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:35.644874 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.644848 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a02173-6525-4899-918a-8087a3b5a45d-kube-api-access-fzrbr" pod "f4a02173-6525-4899-918a-8087a3b5a45d" (UID: "f4a02173-6525-4899-918a-8087a3b5a45d"). InnerVolumeSpecName "kube-api-access-fzrbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:35.744366 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.744336 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzrbr\" (UniqueName: \"kubernetes.io/projected/f4a02173-6525-4899-918a-8087a3b5a45d-kube-api-access-fzrbr\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:35.784511 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.784474 4947 generic.go:356] "Generic (PLEG): container finished" podID="f4a02173-6525-4899-918a-8087a3b5a45d" containerID="91cf28c6446cfc1f7f4abdff2fe2e0fb46ee11bb732c42ff19f4ce2f85651ebe" exitCode=0 Nov 03 16:52:35.784687 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.784525 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z" Nov 03 16:52:35.784687 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.784620 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z" event={"ID":"f4a02173-6525-4899-918a-8087a3b5a45d","Type":"ContainerDied","Data":"91cf28c6446cfc1f7f4abdff2fe2e0fb46ee11bb732c42ff19f4ce2f85651ebe"} Nov 03 16:52:35.784687 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.784668 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z" event={"ID":"f4a02173-6525-4899-918a-8087a3b5a45d","Type":"ContainerDied","Data":"bb8345271dc68296bf9197e7950cfdb23fc50bf40a42e86900d80df6ca99a890"} Nov 03 16:52:35.784875 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.784729 4947 scope.go:122] "RemoveContainer" containerID="91cf28c6446cfc1f7f4abdff2fe2e0fb46ee11bb732c42ff19f4ce2f85651ebe" Nov 03 16:52:35.786619 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.786594 4947 generic.go:356] "Generic (PLEG): container finished" podID="3275cc93-2a98-42d2-9f4a-84ce9c10fbe5" containerID="8727ebbe07cce33501979af62f9149a7476e3751d3bbc469750335af2ae7f92a" exitCode=0 Nov 03 16:52:35.786619 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.786596 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" Nov 03 16:52:35.786958 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.786918 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" event={"ID":"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5","Type":"ContainerDied","Data":"8727ebbe07cce33501979af62f9149a7476e3751d3bbc469750335af2ae7f92a"} Nov 03 16:52:35.787090 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.786970 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq" event={"ID":"3275cc93-2a98-42d2-9f4a-84ce9c10fbe5","Type":"ContainerDied","Data":"29932b659759dec87f3157bd5efce409529001c2d33f681b2f1f3f84f887adca"} Nov 03 16:52:35.790747 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.790724 4947 generic.go:356] "Generic (PLEG): container finished" podID="9d2d985a-3b44-4af4-bb44-6b5faa064127" containerID="6645e982fc8cae00596b6d6922987bc29780ab0524e8d30c2471c15b40702d4e" exitCode=0 Nov 03 16:52:35.790859 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.790808 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26" event={"ID":"9d2d985a-3b44-4af4-bb44-6b5faa064127","Type":"ContainerDied","Data":"6645e982fc8cae00596b6d6922987bc29780ab0524e8d30c2471c15b40702d4e"} Nov 03 16:52:35.790859 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.790846 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26" event={"ID":"9d2d985a-3b44-4af4-bb44-6b5faa064127","Type":"ContainerDied","Data":"4fdc0eb272665ce81592925167b3f1be7882f5f06a9f61d509804159c055248d"} Nov 03 16:52:35.790974 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.790909 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26" Nov 03 16:52:35.795903 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.795874 4947 scope.go:122] "RemoveContainer" containerID="91cf28c6446cfc1f7f4abdff2fe2e0fb46ee11bb732c42ff19f4ce2f85651ebe" Nov 03 16:52:35.796193 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:52:35.796168 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91cf28c6446cfc1f7f4abdff2fe2e0fb46ee11bb732c42ff19f4ce2f85651ebe\": container with ID starting with 91cf28c6446cfc1f7f4abdff2fe2e0fb46ee11bb732c42ff19f4ce2f85651ebe not found: ID does not exist" containerID="91cf28c6446cfc1f7f4abdff2fe2e0fb46ee11bb732c42ff19f4ce2f85651ebe" Nov 03 16:52:35.796263 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.796203 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91cf28c6446cfc1f7f4abdff2fe2e0fb46ee11bb732c42ff19f4ce2f85651ebe"} err="failed to get container status \"91cf28c6446cfc1f7f4abdff2fe2e0fb46ee11bb732c42ff19f4ce2f85651ebe\": rpc error: code = NotFound desc = could not find container \"91cf28c6446cfc1f7f4abdff2fe2e0fb46ee11bb732c42ff19f4ce2f85651ebe\": container with ID starting with 91cf28c6446cfc1f7f4abdff2fe2e0fb46ee11bb732c42ff19f4ce2f85651ebe not found: ID does not exist" Nov 03 16:52:35.796263 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.796220 4947 scope.go:122] "RemoveContainer" containerID="8727ebbe07cce33501979af62f9149a7476e3751d3bbc469750335af2ae7f92a" Nov 03 16:52:35.808187 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.808170 4947 scope.go:122] "RemoveContainer" containerID="8727ebbe07cce33501979af62f9149a7476e3751d3bbc469750335af2ae7f92a" Nov 03 16:52:35.808458 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:52:35.808439 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8727ebbe07cce33501979af62f9149a7476e3751d3bbc469750335af2ae7f92a\": container with ID starting with 8727ebbe07cce33501979af62f9149a7476e3751d3bbc469750335af2ae7f92a not found: ID does not exist" containerID="8727ebbe07cce33501979af62f9149a7476e3751d3bbc469750335af2ae7f92a" Nov 03 16:52:35.808535 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.808462 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8727ebbe07cce33501979af62f9149a7476e3751d3bbc469750335af2ae7f92a"} err="failed to get container status \"8727ebbe07cce33501979af62f9149a7476e3751d3bbc469750335af2ae7f92a\": rpc error: code = NotFound desc = could not find container \"8727ebbe07cce33501979af62f9149a7476e3751d3bbc469750335af2ae7f92a\": container with ID starting with 8727ebbe07cce33501979af62f9149a7476e3751d3bbc469750335af2ae7f92a not found: ID does not exist" Nov 03 16:52:35.808535 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.808478 4947 scope.go:122] "RemoveContainer" containerID="6645e982fc8cae00596b6d6922987bc29780ab0524e8d30c2471c15b40702d4e" Nov 03 16:52:35.817334 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.817319 4947 scope.go:122] "RemoveContainer" containerID="6645e982fc8cae00596b6d6922987bc29780ab0524e8d30c2471c15b40702d4e" Nov 03 16:52:35.817568 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:52:35.817554 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6645e982fc8cae00596b6d6922987bc29780ab0524e8d30c2471c15b40702d4e\": container with ID starting with 6645e982fc8cae00596b6d6922987bc29780ab0524e8d30c2471c15b40702d4e not found: ID does not exist" containerID="6645e982fc8cae00596b6d6922987bc29780ab0524e8d30c2471c15b40702d4e" Nov 03 16:52:35.817609 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.817574 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6645e982fc8cae00596b6d6922987bc29780ab0524e8d30c2471c15b40702d4e"} err="failed to get container status \"6645e982fc8cae00596b6d6922987bc29780ab0524e8d30c2471c15b40702d4e\": rpc error: code = NotFound desc = could not find container \"6645e982fc8cae00596b6d6922987bc29780ab0524e8d30c2471c15b40702d4e\": container with ID starting with 6645e982fc8cae00596b6d6922987bc29780ab0524e8d30c2471c15b40702d4e not found: ID does not exist" Nov 03 16:52:35.842132 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.842108 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq"] Nov 03 16:52:35.846743 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.846719 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-549271-6f6cb66779-m4pwq"] Nov 03 16:52:35.930969 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.928758 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z"] Nov 03 16:52:35.931334 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.931311 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-549271-ffb9c5d46-bw44z"] Nov 03 16:52:35.959564 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.959531 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26"] Nov 03 16:52:35.963458 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.963430 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-549271-698d488ddd-zbq26"] Nov 03 16:52:35.990461 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:35.990435 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb"] Nov 03 16:52:36.026691 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:36.026635 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk"] Nov 03 16:52:36.567122 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:36.567092 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3275cc93-2a98-42d2-9f4a-84ce9c10fbe5" path="/var/lib/kubelet/pods/3275cc93-2a98-42d2-9f4a-84ce9c10fbe5/volumes" Nov 03 16:52:36.567525 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:36.567511 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d2d985a-3b44-4af4-bb44-6b5faa064127" path="/var/lib/kubelet/pods/9d2d985a-3b44-4af4-bb44-6b5faa064127/volumes" Nov 03 16:52:36.567883 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:36.567869 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4a02173-6525-4899-918a-8087a3b5a45d" path="/var/lib/kubelet/pods/f4a02173-6525-4899-918a-8087a3b5a45d/volumes" Nov 03 16:52:36.665837 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:36.665726 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g" podUID="40a2532d-b751-47c7-b625-dc293b948e01" containerName="payload-pre-processing" containerID="cri-o://43226cecce37df2b028413de3c4fb2b2c5d28782b83ab38212967ff126e032a2" gracePeriod=25 Nov 03 16:52:36.845779 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:36.845739 4947 generic.go:356] "Generic (PLEG): container finished" podID="40a2532d-b751-47c7-b625-dc293b948e01" containerID="43226cecce37df2b028413de3c4fb2b2c5d28782b83ab38212967ff126e032a2" exitCode=0 Nov 03 16:52:36.846250 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:36.845876 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g" event={"ID":"40a2532d-b751-47c7-b625-dc293b948e01","Type":"ContainerDied","Data":"43226cecce37df2b028413de3c4fb2b2c5d28782b83ab38212967ff126e032a2"} Nov 03 16:52:37.023335 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:37.023163 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g" Nov 03 16:52:37.105647 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:37.105568 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a2532d-b751-47c7-b625-dc293b948e01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40a2532d-b751-47c7-b625-dc293b948e01-plugins-config-volume\") pod \"40a2532d-b751-47c7-b625-dc293b948e01\" (UID: \"40a2532d-b751-47c7-b625-dc293b948e01\") " Nov 03 16:52:37.105862 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:37.105746 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a2532d-b751-47c7-b625-dc293b948e01-kube-api-access-mvsj2\" (UniqueName: \"kubernetes.io/projected/40a2532d-b751-47c7-b625-dc293b948e01-kube-api-access-mvsj2\") pod \"40a2532d-b751-47c7-b625-dc293b948e01\" (UID: \"40a2532d-b751-47c7-b625-dc293b948e01\") " Nov 03 16:52:37.106017 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:37.105967 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a2532d-b751-47c7-b625-dc293b948e01-plugins-config-volume" pod "40a2532d-b751-47c7-b625-dc293b948e01" (UID: "40a2532d-b751-47c7-b625-dc293b948e01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:37.106369 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:37.106348 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40a2532d-b751-47c7-b625-dc293b948e01-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:37.108884 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:37.108855 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a2532d-b751-47c7-b625-dc293b948e01-kube-api-access-mvsj2" pod "40a2532d-b751-47c7-b625-dc293b948e01" (UID: "40a2532d-b751-47c7-b625-dc293b948e01"). InnerVolumeSpecName "kube-api-access-mvsj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:37.207293 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:37.207262 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvsj2\" (UniqueName: \"kubernetes.io/projected/40a2532d-b751-47c7-b625-dc293b948e01-kube-api-access-mvsj2\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:37.855022 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:37.854989 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g" event={"ID":"40a2532d-b751-47c7-b625-dc293b948e01","Type":"ContainerDied","Data":"8b95ade4d266dc32210d63b63f26abeeb9163204cf128b1b3e0d779761c7597e"} Nov 03 16:52:37.855022 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:37.855018 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g" Nov 03 16:52:37.855448 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:37.855052 4947 scope.go:122] "RemoveContainer" containerID="43226cecce37df2b028413de3c4fb2b2c5d28782b83ab38212967ff126e032a2" Nov 03 16:52:37.922451 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:37.922421 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g"] Nov 03 16:52:37.925022 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:37.925003 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f2923b-7b896d65c5-6td6g"] Nov 03 16:52:38.569259 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:38.569226 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40a2532d-b751-47c7-b625-dc293b948e01" path="/var/lib/kubelet/pods/40a2532d-b751-47c7-b625-dc293b948e01/volumes" Nov 03 16:52:40.916272 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:40.916237 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq"] Nov 03 16:52:40.916835 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:40.916514 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" podUID="f6c68b79-535a-403e-aab9-5a847fd988e8" containerName="maas-api" containerID="cri-o://21fb8a0f4df5ca8c9015f6e7857c5c7b66ea14bdff04a5b5ea71ca2e1c2b6212" gracePeriod=30 Nov 03 16:52:40.990800 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:40.990748 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb" podUID="4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c" containerName="payload-processing" containerID="cri-o://aa07d7233d17e4e3d3b3b46bb47d86c7a7bc5be289046e842026a65887b6480b" gracePeriod=25 Nov 03 16:52:41.027713 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.027664 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk" podUID="51d1aac5-372c-4c1b-9a6e-035abe38e775" containerName="payload-pre-processing" containerID="cri-o://330c76be156249faf0a56dbf00fdf5e280206cda455f0688b77bdf42beb80f0c" gracePeriod=25 Nov 03 16:52:41.366396 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.365973 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" Nov 03 16:52:41.418963 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.418895 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb" Nov 03 16:52:41.436658 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.434885 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk" Nov 03 16:52:41.482933 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.482902 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c68b79-535a-403e-aab9-5a847fd988e8-kube-api-access-mgnrf\" (UniqueName: \"kubernetes.io/projected/f6c68b79-535a-403e-aab9-5a847fd988e8-kube-api-access-mgnrf\") pod \"f6c68b79-535a-403e-aab9-5a847fd988e8\" (UID: \"f6c68b79-535a-403e-aab9-5a847fd988e8\") " Nov 03 16:52:41.483203 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.483185 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d1aac5-372c-4c1b-9a6e-035abe38e775-kube-api-access-4l9mz\" (UniqueName: \"kubernetes.io/projected/51d1aac5-372c-4c1b-9a6e-035abe38e775-kube-api-access-4l9mz\") pod \"51d1aac5-372c-4c1b-9a6e-035abe38e775\" (UID: \"51d1aac5-372c-4c1b-9a6e-035abe38e775\") " Nov 03 16:52:41.483365 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.483351 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6c68b79-535a-403e-aab9-5a847fd988e8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c68b79-535a-403e-aab9-5a847fd988e8-openshift-service-ca\") pod \"f6c68b79-535a-403e-aab9-5a847fd988e8\" (UID: \"f6c68b79-535a-403e-aab9-5a847fd988e8\") " Nov 03 16:52:41.483558 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.483541 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6c68b79-535a-403e-aab9-5a847fd988e8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6c68b79-535a-403e-aab9-5a847fd988e8-metrics-certs\") pod \"f6c68b79-535a-403e-aab9-5a847fd988e8\" (UID: \"f6c68b79-535a-403e-aab9-5a847fd988e8\") " Nov 03 16:52:41.483812 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.483790 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51d1aac5-372c-4c1b-9a6e-035abe38e775-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d1aac5-372c-4c1b-9a6e-035abe38e775-plugins-config-volume\") pod \"51d1aac5-372c-4c1b-9a6e-035abe38e775\" (UID: \"51d1aac5-372c-4c1b-9a6e-035abe38e775\") " Nov 03 16:52:41.484027 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.484013 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6c68b79-535a-403e-aab9-5a847fd988e8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6c68b79-535a-403e-aab9-5a847fd988e8-maas-api-tls\") pod \"f6c68b79-535a-403e-aab9-5a847fd988e8\" (UID: \"f6c68b79-535a-403e-aab9-5a847fd988e8\") " Nov 03 16:52:41.484164 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.484150 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c-kube-api-access-2cqr2\" (UniqueName: \"kubernetes.io/projected/4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c-kube-api-access-2cqr2\") pod \"4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c\" (UID: \"4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c\") " Nov 03 16:52:41.484266 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.484247 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c-plugins-config-volume\") pod \"4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c\" (UID: \"4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c\") " Nov 03 16:52:41.485425 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.485400 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c-plugins-config-volume" pod "4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c" (UID: "4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:41.487880 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.487547 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c68b79-535a-403e-aab9-5a847fd988e8-openshift-service-ca" pod "f6c68b79-535a-403e-aab9-5a847fd988e8" (UID: "f6c68b79-535a-403e-aab9-5a847fd988e8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:41.488479 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.488446 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d1aac5-372c-4c1b-9a6e-035abe38e775-plugins-config-volume" pod "51d1aac5-372c-4c1b-9a6e-035abe38e775" (UID: "51d1aac5-372c-4c1b-9a6e-035abe38e775"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:41.511957 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.508475 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c68b79-535a-403e-aab9-5a847fd988e8-metrics-certs" pod "f6c68b79-535a-403e-aab9-5a847fd988e8" (UID: "f6c68b79-535a-403e-aab9-5a847fd988e8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:41.511957 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.511895 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d1aac5-372c-4c1b-9a6e-035abe38e775-kube-api-access-4l9mz" pod "51d1aac5-372c-4c1b-9a6e-035abe38e775" (UID: "51d1aac5-372c-4c1b-9a6e-035abe38e775"). InnerVolumeSpecName "kube-api-access-4l9mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:41.512295 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.512256 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c-kube-api-access-2cqr2" pod "4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c" (UID: "4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c"). InnerVolumeSpecName "kube-api-access-2cqr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:41.512589 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.512558 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c68b79-535a-403e-aab9-5a847fd988e8-maas-api-tls" pod "f6c68b79-535a-403e-aab9-5a847fd988e8" (UID: "f6c68b79-535a-403e-aab9-5a847fd988e8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:52:41.516404 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.516367 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c68b79-535a-403e-aab9-5a847fd988e8-kube-api-access-mgnrf" pod "f6c68b79-535a-403e-aab9-5a847fd988e8" (UID: "f6c68b79-535a-403e-aab9-5a847fd988e8"). InnerVolumeSpecName "kube-api-access-mgnrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:41.588311 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.588280 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6c68b79-535a-403e-aab9-5a847fd988e8-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:41.588311 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.588310 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51d1aac5-372c-4c1b-9a6e-035abe38e775-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:41.588502 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.588323 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6c68b79-535a-403e-aab9-5a847fd988e8-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:41.588502 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.588332 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cqr2\" (UniqueName: \"kubernetes.io/projected/4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c-kube-api-access-2cqr2\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:41.588502 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.588342 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:41.588502 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.588350 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgnrf\" (UniqueName: \"kubernetes.io/projected/f6c68b79-535a-403e-aab9-5a847fd988e8-kube-api-access-mgnrf\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:41.588502 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.588358 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l9mz\" (UniqueName: \"kubernetes.io/projected/51d1aac5-372c-4c1b-9a6e-035abe38e775-kube-api-access-4l9mz\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:41.588502 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.588372 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6c68b79-535a-403e-aab9-5a847fd988e8-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:41.881559 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.881521 4947 generic.go:356] "Generic (PLEG): container finished" podID="f6c68b79-535a-403e-aab9-5a847fd988e8" containerID="21fb8a0f4df5ca8c9015f6e7857c5c7b66ea14bdff04a5b5ea71ca2e1c2b6212" exitCode=0 Nov 03 16:52:41.881788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.881577 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" Nov 03 16:52:41.881788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.881605 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" event={"ID":"f6c68b79-535a-403e-aab9-5a847fd988e8","Type":"ContainerDied","Data":"21fb8a0f4df5ca8c9015f6e7857c5c7b66ea14bdff04a5b5ea71ca2e1c2b6212"} Nov 03 16:52:41.881788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.881662 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq" event={"ID":"f6c68b79-535a-403e-aab9-5a847fd988e8","Type":"ContainerDied","Data":"60d1800f56d8fe8713960566457b301a3639aec597f171847bf79ef9fba33ca2"} Nov 03 16:52:41.881788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.881690 4947 scope.go:122] "RemoveContainer" containerID="21fb8a0f4df5ca8c9015f6e7857c5c7b66ea14bdff04a5b5ea71ca2e1c2b6212" Nov 03 16:52:41.883231 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.883200 4947 generic.go:356] "Generic (PLEG): container finished" podID="51d1aac5-372c-4c1b-9a6e-035abe38e775" containerID="330c76be156249faf0a56dbf00fdf5e280206cda455f0688b77bdf42beb80f0c" exitCode=0 Nov 03 16:52:41.883483 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.883240 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk" Nov 03 16:52:41.883483 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.883279 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk" event={"ID":"51d1aac5-372c-4c1b-9a6e-035abe38e775","Type":"ContainerDied","Data":"330c76be156249faf0a56dbf00fdf5e280206cda455f0688b77bdf42beb80f0c"} Nov 03 16:52:41.883483 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.883324 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk" event={"ID":"51d1aac5-372c-4c1b-9a6e-035abe38e775","Type":"ContainerDied","Data":"ab046b8e0b07a0ef34e7acf27865ccf8dce741f446557741bdb1ece313fd8a91"} Nov 03 16:52:41.884949 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.884923 4947 generic.go:356] "Generic (PLEG): container finished" podID="4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c" containerID="aa07d7233d17e4e3d3b3b46bb47d86c7a7bc5be289046e842026a65887b6480b" exitCode=0 Nov 03 16:52:41.885078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.884970 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb" Nov 03 16:52:41.885078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.884978 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb" event={"ID":"4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c","Type":"ContainerDied","Data":"aa07d7233d17e4e3d3b3b46bb47d86c7a7bc5be289046e842026a65887b6480b"} Nov 03 16:52:41.885078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.885041 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb" event={"ID":"4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c","Type":"ContainerDied","Data":"abd41a1920420f7dbce67ded070998d1f14c223d2ee4cc20f5ac0cbb9741e880"} Nov 03 16:52:41.894494 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.894298 4947 scope.go:122] "RemoveContainer" containerID="21fb8a0f4df5ca8c9015f6e7857c5c7b66ea14bdff04a5b5ea71ca2e1c2b6212" Nov 03 16:52:41.894629 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:52:41.894603 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21fb8a0f4df5ca8c9015f6e7857c5c7b66ea14bdff04a5b5ea71ca2e1c2b6212\": container with ID starting with 21fb8a0f4df5ca8c9015f6e7857c5c7b66ea14bdff04a5b5ea71ca2e1c2b6212 not found: ID does not exist" containerID="21fb8a0f4df5ca8c9015f6e7857c5c7b66ea14bdff04a5b5ea71ca2e1c2b6212" Nov 03 16:52:41.894686 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.894638 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21fb8a0f4df5ca8c9015f6e7857c5c7b66ea14bdff04a5b5ea71ca2e1c2b6212"} err="failed to get container status \"21fb8a0f4df5ca8c9015f6e7857c5c7b66ea14bdff04a5b5ea71ca2e1c2b6212\": rpc error: code = NotFound desc = could not find container \"21fb8a0f4df5ca8c9015f6e7857c5c7b66ea14bdff04a5b5ea71ca2e1c2b6212\": container with ID starting with 21fb8a0f4df5ca8c9015f6e7857c5c7b66ea14bdff04a5b5ea71ca2e1c2b6212 not found: ID does not exist" Nov 03 16:52:41.894686 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.894656 4947 scope.go:122] "RemoveContainer" containerID="330c76be156249faf0a56dbf00fdf5e280206cda455f0688b77bdf42beb80f0c" Nov 03 16:52:41.903514 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.903499 4947 scope.go:122] "RemoveContainer" containerID="330c76be156249faf0a56dbf00fdf5e280206cda455f0688b77bdf42beb80f0c" Nov 03 16:52:41.903815 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:52:41.903797 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330c76be156249faf0a56dbf00fdf5e280206cda455f0688b77bdf42beb80f0c\": container with ID starting with 330c76be156249faf0a56dbf00fdf5e280206cda455f0688b77bdf42beb80f0c not found: ID does not exist" containerID="330c76be156249faf0a56dbf00fdf5e280206cda455f0688b77bdf42beb80f0c" Nov 03 16:52:41.903875 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.903823 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330c76be156249faf0a56dbf00fdf5e280206cda455f0688b77bdf42beb80f0c"} err="failed to get container status \"330c76be156249faf0a56dbf00fdf5e280206cda455f0688b77bdf42beb80f0c\": rpc error: code = NotFound desc = could not find container \"330c76be156249faf0a56dbf00fdf5e280206cda455f0688b77bdf42beb80f0c\": container with ID starting with 330c76be156249faf0a56dbf00fdf5e280206cda455f0688b77bdf42beb80f0c not found: ID does not exist" Nov 03 16:52:41.903875 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.903838 4947 scope.go:122] "RemoveContainer" containerID="aa07d7233d17e4e3d3b3b46bb47d86c7a7bc5be289046e842026a65887b6480b" Nov 03 16:52:41.915778 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.915758 4947 scope.go:122] "RemoveContainer" containerID="aa07d7233d17e4e3d3b3b46bb47d86c7a7bc5be289046e842026a65887b6480b" Nov 03 16:52:41.916121 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:52:41.916096 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa07d7233d17e4e3d3b3b46bb47d86c7a7bc5be289046e842026a65887b6480b\": container with ID starting with aa07d7233d17e4e3d3b3b46bb47d86c7a7bc5be289046e842026a65887b6480b not found: ID does not exist" containerID="aa07d7233d17e4e3d3b3b46bb47d86c7a7bc5be289046e842026a65887b6480b" Nov 03 16:52:41.916187 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.916128 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa07d7233d17e4e3d3b3b46bb47d86c7a7bc5be289046e842026a65887b6480b"} err="failed to get container status \"aa07d7233d17e4e3d3b3b46bb47d86c7a7bc5be289046e842026a65887b6480b\": rpc error: code = NotFound desc = could not find container \"aa07d7233d17e4e3d3b3b46bb47d86c7a7bc5be289046e842026a65887b6480b\": container with ID starting with aa07d7233d17e4e3d3b3b46bb47d86c7a7bc5be289046e842026a65887b6480b not found: ID does not exist" Nov 03 16:52:41.939541 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.939522 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb"] Nov 03 16:52:41.945546 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.945524 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-9160f5-7b97d5c9dc-797qb"] Nov 03 16:52:41.982541 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.982517 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq"] Nov 03 16:52:41.986092 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:41.986069 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad7bf6-6df996f48f-7b6cq"] Nov 03 16:52:42.034896 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:42.034867 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk"] Nov 03 16:52:42.047447 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:42.047418 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-9160f5-bb44f6d5f-nc2rk"] Nov 03 16:52:42.565718 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:42.565654 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c" path="/var/lib/kubelet/pods/4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c/volumes" Nov 03 16:52:42.566264 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:42.566239 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51d1aac5-372c-4c1b-9a6e-035abe38e775" path="/var/lib/kubelet/pods/51d1aac5-372c-4c1b-9a6e-035abe38e775/volumes" Nov 03 16:52:42.566811 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:42.566792 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6c68b79-535a-403e-aab9-5a847fd988e8" path="/var/lib/kubelet/pods/f6c68b79-535a-403e-aab9-5a847fd988e8/volumes" Nov 03 16:52:50.721111 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:50.721076 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n"] Nov 03 16:52:50.721660 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:50.721279 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" podUID="bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" containerName="istio-proxy" containerID="cri-o://45109a19cfcbe438a052388c0126bf372081960d733e286e6d6969eecdd1b703" gracePeriod=30 Nov 03 16:52:53.327242 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:53.327208 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:52:55.991460 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:55.991438 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:52:56.051078 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.051048 4947 generic.go:356] "Generic (PLEG): container finished" podID="bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" containerID="45109a19cfcbe438a052388c0126bf372081960d733e286e6d6969eecdd1b703" exitCode=0 Nov 03 16:52:56.051279 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.051112 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" Nov 03 16:52:56.051279 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.051147 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" event={"ID":"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159","Type":"ContainerDied","Data":"45109a19cfcbe438a052388c0126bf372081960d733e286e6d6969eecdd1b703"} Nov 03 16:52:56.051279 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.051224 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n" event={"ID":"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159","Type":"ContainerDied","Data":"a154e24428a064660f1bfe23819c62bab2bc9bd3cca29928b7567f8cac08767b"} Nov 03 16:52:56.051279 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.051251 4947 scope.go:122] "RemoveContainer" containerID="45109a19cfcbe438a052388c0126bf372081960d733e286e6d6969eecdd1b703" Nov 03 16:52:56.066410 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.066301 4947 scope.go:122] "RemoveContainer" containerID="eca4fd1e96600bd71ebe9180b7e077788b19a865d43dfb3ad3e86d9d2813f414" Nov 03 16:52:56.076385 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.076364 4947 scope.go:122] "RemoveContainer" containerID="45109a19cfcbe438a052388c0126bf372081960d733e286e6d6969eecdd1b703" Nov 03 16:52:56.076835 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:52:56.076690 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45109a19cfcbe438a052388c0126bf372081960d733e286e6d6969eecdd1b703\": container with ID starting with 45109a19cfcbe438a052388c0126bf372081960d733e286e6d6969eecdd1b703 not found: ID does not exist" containerID="45109a19cfcbe438a052388c0126bf372081960d733e286e6d6969eecdd1b703" Nov 03 16:52:56.076938 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.076846 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45109a19cfcbe438a052388c0126bf372081960d733e286e6d6969eecdd1b703"} err="failed to get container status \"45109a19cfcbe438a052388c0126bf372081960d733e286e6d6969eecdd1b703\": rpc error: code = NotFound desc = could not find container \"45109a19cfcbe438a052388c0126bf372081960d733e286e6d6969eecdd1b703\": container with ID starting with 45109a19cfcbe438a052388c0126bf372081960d733e286e6d6969eecdd1b703 not found: ID does not exist" Nov 03 16:52:56.076938 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.076865 4947 scope.go:122] "RemoveContainer" containerID="eca4fd1e96600bd71ebe9180b7e077788b19a865d43dfb3ad3e86d9d2813f414" Nov 03 16:52:56.077189 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:52:56.077162 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eca4fd1e96600bd71ebe9180b7e077788b19a865d43dfb3ad3e86d9d2813f414\": container with ID starting with eca4fd1e96600bd71ebe9180b7e077788b19a865d43dfb3ad3e86d9d2813f414 not found: ID does not exist" containerID="eca4fd1e96600bd71ebe9180b7e077788b19a865d43dfb3ad3e86d9d2813f414" Nov 03 16:52:56.077241 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.077195 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca4fd1e96600bd71ebe9180b7e077788b19a865d43dfb3ad3e86d9d2813f414"} err="failed to get container status \"eca4fd1e96600bd71ebe9180b7e077788b19a865d43dfb3ad3e86d9d2813f414\": rpc error: code = NotFound desc = could not find container \"eca4fd1e96600bd71ebe9180b7e077788b19a865d43dfb3ad3e86d9d2813f414\": container with ID starting with eca4fd1e96600bd71ebe9180b7e077788b19a865d43dfb3ad3e86d9d2813f414 not found: ID does not exist" Nov 03 16:52:56.088981 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.088957 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-workload-certs\") pod \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " Nov 03 16:52:56.089101 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.089023 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-workload-socket\") pod \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " Nov 03 16:52:56.089101 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.089048 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istiod-ca-cert\") pod \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " Nov 03 16:52:56.089101 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.089093 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-credential-socket\") pod \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " Nov 03 16:52:56.089270 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.089139 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-data\") pod \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " Nov 03 16:52:56.089270 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.089192 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-kube-api-access-sdplm\" (UniqueName: \"kubernetes.io/projected/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-kube-api-access-sdplm\") pod \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " Nov 03 16:52:56.089270 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.089224 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-envoy\") pod \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " Nov 03 16:52:56.089270 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.089246 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-workload-certs" pod "bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" (UID: "bfe8beeb-2b56-4b2d-8d21-51c2ce71f159"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:56.089469 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.089253 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-token\" (UniqueName: \"kubernetes.io/projected/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-token\") pod \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " Nov 03 16:52:56.089469 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.089347 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-podinfo\") pod \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\" (UID: \"bfe8beeb-2b56-4b2d-8d21-51c2ce71f159\") " Nov 03 16:52:56.089587 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.089563 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-credential-socket" pod "bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" (UID: "bfe8beeb-2b56-4b2d-8d21-51c2ce71f159"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:56.090141 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.089966 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:56.090141 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.089990 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:56.090298 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.090215 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-workload-socket" pod "bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" (UID: "bfe8beeb-2b56-4b2d-8d21-51c2ce71f159"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:56.090362 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.090330 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istiod-ca-cert" pod "bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" (UID: "bfe8beeb-2b56-4b2d-8d21-51c2ce71f159"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:52:56.090543 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.090466 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-data" pod "bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" (UID: "bfe8beeb-2b56-4b2d-8d21-51c2ce71f159"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:56.092121 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.092100 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-token" pod "bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" (UID: "bfe8beeb-2b56-4b2d-8d21-51c2ce71f159"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:56.092236 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.092098 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-podinfo" pod "bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" (UID: "bfe8beeb-2b56-4b2d-8d21-51c2ce71f159"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:52:56.092653 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.092629 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-envoy" pod "bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" (UID: "bfe8beeb-2b56-4b2d-8d21-51c2ce71f159"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:52:56.093208 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.093189 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-kube-api-access-sdplm" pod "bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" (UID: "bfe8beeb-2b56-4b2d-8d21-51c2ce71f159"). InnerVolumeSpecName "kube-api-access-sdplm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:52:56.191464 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.191432 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:56.191610 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.191460 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:56.191610 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.191504 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:56.191610 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.191513 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdplm\" (UniqueName: \"kubernetes.io/projected/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-kube-api-access-sdplm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:56.191610 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.191524 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:56.191610 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.191533 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:56.191610 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.191541 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:52:56.403277 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.403241 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n"] Nov 03 16:52:56.416391 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.416366 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ad7bf6-openshift-default-769f6d4fc-jhg2n"] Nov 03 16:52:56.565527 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:56.565497 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" path="/var/lib/kubelet/pods/bfe8beeb-2b56-4b2d-8d21-51c2ce71f159/volumes" Nov 03 16:52:58.291178 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291148 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m"] Nov 03 16:52:58.291655 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291642 4947 cpu_manager.go:414] "containerMap: removing container" podUID="a707604b-dc82-4c5b-bdc4-9f88027df05c" containerName="istio-proxy" Nov 03 16:52:58.291730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291656 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a707604b-dc82-4c5b-bdc4-9f88027df05c" containerName="istio-proxy" Nov 03 16:52:58.291730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291679 4947 cpu_manager.go:414] "containerMap: removing container" podUID="007c9802-d9bf-4cd2-a062-6da9b70c31ff" containerName="payload-pre-processing" Nov 03 16:52:58.291730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291684 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007c9802-d9bf-4cd2-a062-6da9b70c31ff" containerName="payload-pre-processing" Nov 03 16:52:58.291730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291691 4947 cpu_manager.go:414] "containerMap: removing container" podUID="ff88d741-c433-40a3-804b-24742f4a8140" containerName="payload-processing" Nov 03 16:52:58.291730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291710 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff88d741-c433-40a3-804b-24742f4a8140" containerName="payload-processing" Nov 03 16:52:58.291730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291718 4947 cpu_manager.go:414] "containerMap: removing container" podUID="a0adbe8a-947c-433a-886e-c824cc5d0566" containerName="maas-api" Nov 03 16:52:58.291730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291722 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0adbe8a-947c-433a-886e-c824cc5d0566" containerName="maas-api" Nov 03 16:52:58.291730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291731 4947 cpu_manager.go:414] "containerMap: removing container" podUID="bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" containerName="istio-proxy" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291735 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" containerName="istio-proxy" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291742 4947 cpu_manager.go:414] "containerMap: removing container" podUID="40a2532d-b751-47c7-b625-dc293b948e01" containerName="payload-pre-processing" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291746 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a2532d-b751-47c7-b625-dc293b948e01" containerName="payload-pre-processing" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291757 4947 cpu_manager.go:414] "containerMap: removing container" podUID="f4a02173-6525-4899-918a-8087a3b5a45d" containerName="payload-pre-processing" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291761 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a02173-6525-4899-918a-8087a3b5a45d" containerName="payload-pre-processing" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291773 4947 cpu_manager.go:414] "containerMap: removing container" podUID="3275cc93-2a98-42d2-9f4a-84ce9c10fbe5" containerName="maas-api" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291777 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3275cc93-2a98-42d2-9f4a-84ce9c10fbe5" containerName="maas-api" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291784 4947 cpu_manager.go:414] "containerMap: removing container" podUID="a707604b-dc82-4c5b-bdc4-9f88027df05c" containerName="istio-proxy" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291788 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a707604b-dc82-4c5b-bdc4-9f88027df05c" containerName="istio-proxy" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291796 4947 cpu_manager.go:414] "containerMap: removing container" podUID="bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" containerName="istio-proxy" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291801 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" containerName="istio-proxy" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291807 4947 cpu_manager.go:414] "containerMap: removing container" podUID="f6c68b79-535a-403e-aab9-5a847fd988e8" containerName="maas-api" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291811 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c68b79-535a-403e-aab9-5a847fd988e8" containerName="maas-api" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291827 4947 cpu_manager.go:414] "containerMap: removing container" podUID="51d1aac5-372c-4c1b-9a6e-035abe38e775" containerName="payload-pre-processing" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291831 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d1aac5-372c-4c1b-9a6e-035abe38e775" containerName="payload-pre-processing" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291836 4947 cpu_manager.go:414] "containerMap: removing container" podUID="4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c" containerName="payload-processing" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291840 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c" containerName="payload-processing" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291845 4947 cpu_manager.go:414] "containerMap: removing container" podUID="9d2d985a-3b44-4af4-bb44-6b5faa064127" containerName="payload-processing" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291850 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d2d985a-3b44-4af4-bb44-6b5faa064127" containerName="payload-processing" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291921 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0adbe8a-947c-433a-886e-c824cc5d0566" containerName="maas-api" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291929 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" containerName="istio-proxy" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291937 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a2532d-b751-47c7-b625-dc293b948e01" containerName="payload-pre-processing" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291943 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a02173-6525-4899-918a-8087a3b5a45d" containerName="payload-pre-processing" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291951 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="3275cc93-2a98-42d2-9f4a-84ce9c10fbe5" containerName="maas-api" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291957 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="a707604b-dc82-4c5b-bdc4-9f88027df05c" containerName="istio-proxy" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291962 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe8beeb-2b56-4b2d-8d21-51c2ce71f159" containerName="istio-proxy" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291969 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c68b79-535a-403e-aab9-5a847fd988e8" containerName="maas-api" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291975 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d1aac5-372c-4c1b-9a6e-035abe38e775" containerName="payload-pre-processing" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291980 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb5aaaa-b117-46b7-8f42-2c4ec6cffe6c" containerName="payload-processing" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291984 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d2d985a-3b44-4af4-bb44-6b5faa064127" containerName="payload-processing" Nov 03 16:52:58.291982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291992 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="007c9802-d9bf-4cd2-a062-6da9b70c31ff" containerName="payload-pre-processing" Nov 03 16:52:58.293232 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.291998 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff88d741-c433-40a3-804b-24742f4a8140" containerName="payload-processing" Nov 03 16:52:58.297105 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.297086 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m" Nov 03 16:52:58.322466 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.322429 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-be8495\"" Nov 03 16:52:58.326503 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.326473 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m"] Nov 03 16:52:58.341438 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.341407 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67bmv\" (UniqueName: \"kubernetes.io/projected/694cfc14-a5d4-4111-9266-fe195921dfc7-kube-api-access-67bmv\") pod \"payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m\" (UID: \"694cfc14-a5d4-4111-9266-fe195921dfc7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m" Nov 03 16:52:58.341608 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.341499 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694cfc14-a5d4-4111-9266-fe195921dfc7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m\" (UID: \"694cfc14-a5d4-4111-9266-fe195921dfc7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m" Nov 03 16:52:58.367234 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.367195 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx"] Nov 03 16:52:58.367988 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.367967 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="a707604b-dc82-4c5b-bdc4-9f88027df05c" containerName="istio-proxy" Nov 03 16:52:58.371504 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.371487 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx" Nov 03 16:52:58.392768 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.392741 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-be8495-dockercfg-dq9px\"" Nov 03 16:52:58.394966 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.394943 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx"] Nov 03 16:52:58.496719 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.493906 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c2897a-3db9-4dfb-9063-8f67092cd30e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx\" (UID: \"21c2897a-3db9-4dfb-9063-8f67092cd30e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx" Nov 03 16:52:58.496719 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.493966 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95rsw\" (UniqueName: \"kubernetes.io/projected/21c2897a-3db9-4dfb-9063-8f67092cd30e-kube-api-access-95rsw\") pod \"payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx\" (UID: \"21c2897a-3db9-4dfb-9063-8f67092cd30e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx" Nov 03 16:52:58.496719 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.494048 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67bmv\" (UniqueName: \"kubernetes.io/projected/694cfc14-a5d4-4111-9266-fe195921dfc7-kube-api-access-67bmv\") pod \"payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m\" (UID: \"694cfc14-a5d4-4111-9266-fe195921dfc7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m" Nov 03 16:52:58.496719 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.494130 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694cfc14-a5d4-4111-9266-fe195921dfc7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m\" (UID: \"694cfc14-a5d4-4111-9266-fe195921dfc7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m" Nov 03 16:52:58.496719 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.495121 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694cfc14-a5d4-4111-9266-fe195921dfc7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m\" (UID: \"694cfc14-a5d4-4111-9266-fe195921dfc7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m" Nov 03 16:52:58.567774 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.566725 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67bmv\" (UniqueName: \"kubernetes.io/projected/694cfc14-a5d4-4111-9266-fe195921dfc7-kube-api-access-67bmv\") pod \"payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m\" (UID: \"694cfc14-a5d4-4111-9266-fe195921dfc7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m" Nov 03 16:52:58.596783 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.596741 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c2897a-3db9-4dfb-9063-8f67092cd30e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx\" (UID: \"21c2897a-3db9-4dfb-9063-8f67092cd30e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx" Nov 03 16:52:58.596990 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.596798 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95rsw\" (UniqueName: \"kubernetes.io/projected/21c2897a-3db9-4dfb-9063-8f67092cd30e-kube-api-access-95rsw\") pod \"payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx\" (UID: \"21c2897a-3db9-4dfb-9063-8f67092cd30e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx" Nov 03 16:52:58.598113 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.598089 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c2897a-3db9-4dfb-9063-8f67092cd30e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx\" (UID: \"21c2897a-3db9-4dfb-9063-8f67092cd30e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx" Nov 03 16:52:58.611601 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.611577 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m" Nov 03 16:52:58.621857 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.621832 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95rsw\" (UniqueName: \"kubernetes.io/projected/21c2897a-3db9-4dfb-9063-8f67092cd30e-kube-api-access-95rsw\") pod \"payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx\" (UID: \"21c2897a-3db9-4dfb-9063-8f67092cd30e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx" Nov 03 16:52:58.684860 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.684827 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx" Nov 03 16:52:58.927179 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.927149 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m"] Nov 03 16:52:58.928148 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:52:58.928119 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694cfc14_a5d4_4111_9266_fe195921dfc7.slice/crio-e2e3de1ea5bc4a4977265ac8529b2282214ce82c83fba47c6c5ce3d9dfbe5fb1 WatchSource:0}: Error finding container e2e3de1ea5bc4a4977265ac8529b2282214ce82c83fba47c6c5ce3d9dfbe5fb1: Status 404 returned error can't find the container with id e2e3de1ea5bc4a4977265ac8529b2282214ce82c83fba47c6c5ce3d9dfbe5fb1 Nov 03 16:52:58.992958 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:52:58.992934 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21c2897a_3db9_4dfb_9063_8f67092cd30e.slice/crio-eb3c1ea12ea6661c6caa4be90dbb08dd0de9ff9a7825b242784c221e16c6f4f2 WatchSource:0}: Error finding container eb3c1ea12ea6661c6caa4be90dbb08dd0de9ff9a7825b242784c221e16c6f4f2: Status 404 returned error can't find the container with id eb3c1ea12ea6661c6caa4be90dbb08dd0de9ff9a7825b242784c221e16c6f4f2 Nov 03 16:52:58.994652 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:58.994628 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx"] Nov 03 16:52:59.082373 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:59.082338 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx" event={"ID":"21c2897a-3db9-4dfb-9063-8f67092cd30e","Type":"ContainerStarted","Data":"3b151e9ca589089b47e1d755711bca5cf7e427c0ffeb7575298906acaf7efc06"} Nov 03 16:52:59.082508 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:59.082402 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx" event={"ID":"21c2897a-3db9-4dfb-9063-8f67092cd30e","Type":"ContainerStarted","Data":"eb3c1ea12ea6661c6caa4be90dbb08dd0de9ff9a7825b242784c221e16c6f4f2"} Nov 03 16:52:59.082508 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:59.082432 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx" Nov 03 16:52:59.084029 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:59.084006 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m" event={"ID":"694cfc14-a5d4-4111-9266-fe195921dfc7","Type":"ContainerStarted","Data":"e461b8c6c3936a9a8af4e6446cb097f2872ff78bf3c0668ce892f22b2776f69d"} Nov 03 16:52:59.084153 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:59.084059 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m" Nov 03 16:52:59.084153 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:59.084082 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m" event={"ID":"694cfc14-a5d4-4111-9266-fe195921dfc7","Type":"ContainerStarted","Data":"e2e3de1ea5bc4a4977265ac8529b2282214ce82c83fba47c6c5ce3d9dfbe5fb1"} Nov 03 16:52:59.127236 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:59.127184 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx" podStartSLOduration=1.127169356 podStartE2EDuration="1.127169356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:52:59.125753217 +0000 UTC m=+2322.714753187" watchObservedRunningTime="2026-11-03 16:52:59.127169356 +0000 UTC m=+2322.716169323" Nov 03 16:52:59.170461 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:52:59.170415 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m" podStartSLOduration=1.170402673 podStartE2EDuration="1.170402673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:52:59.1682442 +0000 UTC m=+2322.757244169" watchObservedRunningTime="2026-11-03 16:52:59.170402673 +0000 UTC m=+2322.759402640" Nov 03 16:53:00.169395 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:53:00.169360 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:53:06.415428 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:06.415395 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd"] Nov 03 16:53:06.419785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:06.419765 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" Nov 03 16:53:06.448995 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:06.448969 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-be8495-serving-cert\"" Nov 03 16:53:06.453139 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:06.453110 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd"] Nov 03 16:53:06.513774 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:06.513744 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9zkn\" (UniqueName: \"kubernetes.io/projected/eb2f23df-d970-4f16-8292-569a9356374c-kube-api-access-f9zkn\") pod \"maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd\" (UID: \"eb2f23df-d970-4f16-8292-569a9356374c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" Nov 03 16:53:06.513940 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:06.513855 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb2f23df-d970-4f16-8292-569a9356374c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd\" (UID: \"eb2f23df-d970-4f16-8292-569a9356374c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" Nov 03 16:53:06.513940 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:06.513906 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb2f23df-d970-4f16-8292-569a9356374c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd\" (UID: \"eb2f23df-d970-4f16-8292-569a9356374c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" Nov 03 16:53:06.514027 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:06.513949 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb2f23df-d970-4f16-8292-569a9356374c-metrics-certs\") pod \"maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd\" (UID: \"eb2f23df-d970-4f16-8292-569a9356374c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" Nov 03 16:53:06.620720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:06.616920 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb2f23df-d970-4f16-8292-569a9356374c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd\" (UID: \"eb2f23df-d970-4f16-8292-569a9356374c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" Nov 03 16:53:06.620720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:06.617000 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb2f23df-d970-4f16-8292-569a9356374c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd\" (UID: \"eb2f23df-d970-4f16-8292-569a9356374c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" Nov 03 16:53:06.620720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:06.617066 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb2f23df-d970-4f16-8292-569a9356374c-metrics-certs\") pod \"maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd\" (UID: \"eb2f23df-d970-4f16-8292-569a9356374c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" Nov 03 16:53:06.620720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:06.617280 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9zkn\" (UniqueName: \"kubernetes.io/projected/eb2f23df-d970-4f16-8292-569a9356374c-kube-api-access-f9zkn\") pod \"maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd\" (UID: \"eb2f23df-d970-4f16-8292-569a9356374c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" Nov 03 16:53:06.637724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:06.629316 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb2f23df-d970-4f16-8292-569a9356374c-metrics-certs\") pod \"maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd\" (UID: \"eb2f23df-d970-4f16-8292-569a9356374c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" Nov 03 16:53:06.644722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:06.639582 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb2f23df-d970-4f16-8292-569a9356374c-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd\" (UID: \"eb2f23df-d970-4f16-8292-569a9356374c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" Nov 03 16:53:06.670142 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:06.670069 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb2f23df-d970-4f16-8292-569a9356374c-maas-api-tls\") pod \"maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd\" (UID: \"eb2f23df-d970-4f16-8292-569a9356374c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" Nov 03 16:53:06.672234 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:06.672202 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9zkn\" (UniqueName: \"kubernetes.io/projected/eb2f23df-d970-4f16-8292-569a9356374c-kube-api-access-f9zkn\") pod \"maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd\" (UID: \"eb2f23df-d970-4f16-8292-569a9356374c\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" Nov 03 16:53:06.733797 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:06.733769 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" Nov 03 16:53:07.051096 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:07.050938 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd"] Nov 03 16:53:07.051278 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:53:07.051254 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2f23df_d970_4f16_8292_569a9356374c.slice/crio-32cefc6acf0de3f890c48108ad53db25eb5319c7f4d33991c53a867a45b5c674 WatchSource:0}: Error finding container 32cefc6acf0de3f890c48108ad53db25eb5319c7f4d33991c53a867a45b5c674: Status 404 returned error can't find the container with id 32cefc6acf0de3f890c48108ad53db25eb5319c7f4d33991c53a867a45b5c674 Nov 03 16:53:07.175858 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:07.175823 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" event={"ID":"eb2f23df-d970-4f16-8292-569a9356374c","Type":"ContainerStarted","Data":"32cefc6acf0de3f890c48108ad53db25eb5319c7f4d33991c53a867a45b5c674"} Nov 03 16:53:08.184322 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:08.184262 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" event={"ID":"eb2f23df-d970-4f16-8292-569a9356374c","Type":"ContainerStarted","Data":"aea408c87a68e6d37c75b34d9042cd1dcec7cc0ec9a5b1f69c3ce717899cff40"} Nov 03 16:53:08.184774 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:08.184347 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" Nov 03 16:53:08.243462 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:08.243416 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" podStartSLOduration=1.604988864 podStartE2EDuration="2.243403388s" totalImagesPullingTime="638.414524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:08.241783539 +0000 UTC m=+2331.830783506" watchObservedRunningTime="2026-11-03 16:53:08.243403388 +0000 UTC m=+2331.832403354" Nov 03 16:53:08.804267 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:08.804235 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6"] Nov 03 16:53:08.804451 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:08.804429 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" podUID="c802c606-eb7c-497f-a219-a4fc5ef34cd1" containerName="istio-proxy" containerID="cri-o://5535f69d708bf7a355ab210abcd2b1bf3e86f020e623dfb569827f81c1290272" gracePeriod=30 Nov 03 16:53:10.137567 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:10.137540 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m" Nov 03 16:53:10.138010 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:10.137613 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx" Nov 03 16:53:14.077265 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.077209 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:53:14.170710 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.170659 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-podinfo\") pod \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " Nov 03 16:53:14.170934 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.170745 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-workload-socket\") pod \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " Nov 03 16:53:14.170934 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.170784 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istiod-ca-cert\") pod \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " Nov 03 16:53:14.170934 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.170824 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-credential-socket\") pod \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " Nov 03 16:53:14.170934 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.170927 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-envoy\") pod \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " Nov 03 16:53:14.171235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.171010 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-data\") pod \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " Nov 03 16:53:14.171235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.171099 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-token\" (UniqueName: \"kubernetes.io/projected/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-token\") pod \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " Nov 03 16:53:14.171235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.171106 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-workload-socket" pod "c802c606-eb7c-497f-a219-a4fc5ef34cd1" (UID: "c802c606-eb7c-497f-a219-a4fc5ef34cd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:53:14.171235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.171163 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-credential-socket" pod "c802c606-eb7c-497f-a219-a4fc5ef34cd1" (UID: "c802c606-eb7c-497f-a219-a4fc5ef34cd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:53:14.171235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.171213 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c802c606-eb7c-497f-a219-a4fc5ef34cd1-kube-api-access-kpldk\" (UniqueName: \"kubernetes.io/projected/c802c606-eb7c-497f-a219-a4fc5ef34cd1-kube-api-access-kpldk\") pod \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " Nov 03 16:53:14.171488 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.171261 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-workload-certs\") pod \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\" (UID: \"c802c606-eb7c-497f-a219-a4fc5ef34cd1\") " Nov 03 16:53:14.171488 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.171348 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-data" pod "c802c606-eb7c-497f-a219-a4fc5ef34cd1" (UID: "c802c606-eb7c-497f-a219-a4fc5ef34cd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:53:14.171726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.171682 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-workload-certs" pod "c802c606-eb7c-497f-a219-a4fc5ef34cd1" (UID: "c802c606-eb7c-497f-a219-a4fc5ef34cd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:53:14.172175 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.172155 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:53:14.172243 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.172184 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:53:14.172243 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.172199 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:53:14.172243 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.172212 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:53:14.172519 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.172497 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istiod-ca-cert" pod "c802c606-eb7c-497f-a219-a4fc5ef34cd1" (UID: "c802c606-eb7c-497f-a219-a4fc5ef34cd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:53:14.173808 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.173782 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-podinfo" pod "c802c606-eb7c-497f-a219-a4fc5ef34cd1" (UID: "c802c606-eb7c-497f-a219-a4fc5ef34cd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:53:14.174028 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.173998 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-token" pod "c802c606-eb7c-497f-a219-a4fc5ef34cd1" (UID: "c802c606-eb7c-497f-a219-a4fc5ef34cd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:14.174295 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.174272 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c802c606-eb7c-497f-a219-a4fc5ef34cd1-kube-api-access-kpldk" pod "c802c606-eb7c-497f-a219-a4fc5ef34cd1" (UID: "c802c606-eb7c-497f-a219-a4fc5ef34cd1"). InnerVolumeSpecName "kube-api-access-kpldk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:14.175846 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.175819 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-envoy" pod "c802c606-eb7c-497f-a219-a4fc5ef34cd1" (UID: "c802c606-eb7c-497f-a219-a4fc5ef34cd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:53:14.196615 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.196594 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" Nov 03 16:53:14.225775 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.225386 4947 generic.go:356] "Generic (PLEG): container finished" podID="c802c606-eb7c-497f-a219-a4fc5ef34cd1" containerID="5535f69d708bf7a355ab210abcd2b1bf3e86f020e623dfb569827f81c1290272" exitCode=0 Nov 03 16:53:14.225775 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.225495 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" event={"ID":"c802c606-eb7c-497f-a219-a4fc5ef34cd1","Type":"ContainerDied","Data":"5535f69d708bf7a355ab210abcd2b1bf3e86f020e623dfb569827f81c1290272"} Nov 03 16:53:14.225775 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.225581 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" event={"ID":"c802c606-eb7c-497f-a219-a4fc5ef34cd1","Type":"ContainerDied","Data":"7fb3c92304bc1c5239154f408095f6fed89fe7b95513a8af2781194ccb48e032"} Nov 03 16:53:14.225775 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.225636 4947 scope.go:122] "RemoveContainer" containerID="5535f69d708bf7a355ab210abcd2b1bf3e86f020e623dfb569827f81c1290272" Nov 03 16:53:14.226069 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.225864 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6" Nov 03 16:53:14.238434 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.238408 4947 scope.go:122] "RemoveContainer" containerID="5535f69d708bf7a355ab210abcd2b1bf3e86f020e623dfb569827f81c1290272" Nov 03 16:53:14.238686 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:53:14.238669 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5535f69d708bf7a355ab210abcd2b1bf3e86f020e623dfb569827f81c1290272\": container with ID starting with 5535f69d708bf7a355ab210abcd2b1bf3e86f020e623dfb569827f81c1290272 not found: ID does not exist" containerID="5535f69d708bf7a355ab210abcd2b1bf3e86f020e623dfb569827f81c1290272" Nov 03 16:53:14.238753 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.238707 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5535f69d708bf7a355ab210abcd2b1bf3e86f020e623dfb569827f81c1290272"} err="failed to get container status \"5535f69d708bf7a355ab210abcd2b1bf3e86f020e623dfb569827f81c1290272\": rpc error: code = NotFound desc = could not find container \"5535f69d708bf7a355ab210abcd2b1bf3e86f020e623dfb569827f81c1290272\": container with ID starting with 5535f69d708bf7a355ab210abcd2b1bf3e86f020e623dfb569827f81c1290272 not found: ID does not exist" Nov 03 16:53:14.274479 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.274454 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:53:14.274479 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.274477 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:53:14.274675 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.274489 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpldk\" (UniqueName: \"kubernetes.io/projected/c802c606-eb7c-497f-a219-a4fc5ef34cd1-kube-api-access-kpldk\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:53:14.274675 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.274501 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:53:14.274675 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.274509 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c802c606-eb7c-497f-a219-a4fc5ef34cd1-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:53:14.382825 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.382745 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6"] Nov 03 16:53:14.390106 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.390081 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-l9cj6"] Nov 03 16:53:14.472421 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.472383 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:53:14.565016 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:14.564985 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c802c606-eb7c-497f-a219-a4fc5ef34cd1" path="/var/lib/kubelet/pods/c802c606-eb7c-497f-a219-a4fc5ef34cd1/volumes" Nov 03 16:53:19.688616 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.688579 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p"] Nov 03 16:53:19.689367 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.689350 4947 cpu_manager.go:414] "containerMap: removing container" podUID="c802c606-eb7c-497f-a219-a4fc5ef34cd1" containerName="istio-proxy" Nov 03 16:53:19.689424 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.689370 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c802c606-eb7c-497f-a219-a4fc5ef34cd1" containerName="istio-proxy" Nov 03 16:53:19.689521 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.689512 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="c802c606-eb7c-497f-a219-a4fc5ef34cd1" containerName="istio-proxy" Nov 03 16:53:19.695208 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.695186 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.716973 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.716941 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.717112 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.717024 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.717192 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.717111 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwktw\" (UniqueName: \"kubernetes.io/projected/b720f231-ec3b-4e94-bbf5-8920f8976a9e-kube-api-access-fwktw\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.717192 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.717155 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.717316 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.717297 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.717395 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.717377 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.717515 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.717499 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.717583 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.717567 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.717664 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.717640 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.730054 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.729680 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p"] Nov 03 16:53:19.819003 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.818968 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.819172 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.819035 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.819172 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.819064 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.819172 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.819097 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.819324 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.819298 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.819374 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.819357 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwktw\" (UniqueName: \"kubernetes.io/projected/b720f231-ec3b-4e94-bbf5-8920f8976a9e-kube-api-access-fwktw\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.819431 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.819399 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.819482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.819451 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-data\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.819539 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.819497 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-workload-certs\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.819539 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.819504 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.819653 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.819557 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.819807 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.819781 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-workload-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.819929 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.819863 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-credential-socket\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.820094 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.820075 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istiod-ca-cert\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.821867 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.821842 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-podinfo\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.821998 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.821980 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-envoy\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.845911 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.845888 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-token\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:19.846179 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:19.846155 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwktw\" (UniqueName: \"kubernetes.io/projected/b720f231-ec3b-4e94-bbf5-8920f8976a9e-kube-api-access-fwktw\") pod \"e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:20.013162 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:20.013131 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:20.334123 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:20.334095 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p"] Nov 03 16:53:20.334610 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:53:20.334574 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb720f231_ec3b_4e94_bbf5_8920f8976a9e.slice/crio-e87a26c86659f97049de5f3df8da3cf37986804329d1237caaa6c1ce32c44ee0 WatchSource:0}: Error finding container e87a26c86659f97049de5f3df8da3cf37986804329d1237caaa6c1ce32c44ee0: Status 404 returned error can't find the container with id e87a26c86659f97049de5f3df8da3cf37986804329d1237caaa6c1ce32c44ee0 Nov 03 16:53:20.337828 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:20.337797 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:53:20.337911 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:20.337865 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:53:20.337911 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:20.337895 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:53:21.323045 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:21.323010 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" event={"ID":"b720f231-ec3b-4e94-bbf5-8920f8976a9e","Type":"ContainerStarted","Data":"b94f168503075af39a394b9607c2fe9e4f9062f20ae29b4075ad5a9bb230f286"} Nov 03 16:53:21.323671 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:21.323646 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" event={"ID":"b720f231-ec3b-4e94-bbf5-8920f8976a9e","Type":"ContainerStarted","Data":"e87a26c86659f97049de5f3df8da3cf37986804329d1237caaa6c1ce32c44ee0"} Nov 03 16:53:21.386085 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:21.386027 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" podStartSLOduration=2.386012585 podStartE2EDuration="2.386012585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:21.382793415 +0000 UTC m=+2344.971793382" watchObservedRunningTime="2026-11-03 16:53:21.386012585 +0000 UTC m=+2344.975012552" Nov 03 16:53:22.019724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:22.019334 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:22.027586 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:22.026825 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 16:53:22.027586 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:22.026886 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" podUID="b720f231-ec3b-4e94-bbf5-8920f8976a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 16:53:23.016132 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:23.016094 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 16:53:23.018319 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:23.018284 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" podUID="b720f231-ec3b-4e94-bbf5-8920f8976a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 16:53:24.028725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:24.013388 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 16:53:24.028725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:24.013457 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" podUID="b720f231-ec3b-4e94-bbf5-8920f8976a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 16:53:25.019724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:25.019647 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 16:53:25.019945 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:25.019744 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" podUID="b720f231-ec3b-4e94-bbf5-8920f8976a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 16:53:26.023863 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:26.023820 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 16:53:26.024370 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:26.023898 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" podUID="b720f231-ec3b-4e94-bbf5-8920f8976a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 16:53:27.013616 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:27.013580 4947 patch_prober.go:29] interesting pod/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body= Nov 03 16:53:27.013842 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:27.013637 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" podUID="b720f231-ec3b-4e94-bbf5-8920f8976a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" Nov 03 16:53:28.049183 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:28.049148 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:28.050706 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:28.050676 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:28.051948 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:28.051764 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 16:53:30.137599 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:53:30.137563 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:53:31.400477 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:31.400443 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs"] Nov 03 16:53:31.405912 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:31.405893 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs" Nov 03 16:53:31.441961 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:31.441935 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-a11d94\"" Nov 03 16:53:31.442084 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:31.441935 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-a11d94-dockercfg-vjm42\"" Nov 03 16:53:31.444803 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:31.444785 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs"] Nov 03 16:53:31.448734 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:31.448680 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef532337-ef30-48e6-a371-120051ae7ee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs\" (UID: \"ef532337-ef30-48e6-a371-120051ae7ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs" Nov 03 16:53:31.448938 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:31.448921 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd6wj\" (UniqueName: \"kubernetes.io/projected/ef532337-ef30-48e6-a371-120051ae7ee0-kube-api-access-hd6wj\") pod \"payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs\" (UID: \"ef532337-ef30-48e6-a371-120051ae7ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs" Nov 03 16:53:31.551721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:31.551462 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef532337-ef30-48e6-a371-120051ae7ee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs\" (UID: \"ef532337-ef30-48e6-a371-120051ae7ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs" Nov 03 16:53:31.551721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:31.551661 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd6wj\" (UniqueName: \"kubernetes.io/projected/ef532337-ef30-48e6-a371-120051ae7ee0-kube-api-access-hd6wj\") pod \"payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs\" (UID: \"ef532337-ef30-48e6-a371-120051ae7ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs" Nov 03 16:53:31.553013 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:31.552985 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef532337-ef30-48e6-a371-120051ae7ee0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs\" (UID: \"ef532337-ef30-48e6-a371-120051ae7ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs" Nov 03 16:53:31.642713 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:31.642654 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd6wj\" (UniqueName: \"kubernetes.io/projected/ef532337-ef30-48e6-a371-120051ae7ee0-kube-api-access-hd6wj\") pod \"payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs\" (UID: \"ef532337-ef30-48e6-a371-120051ae7ee0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs" Nov 03 16:53:31.719860 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:31.719791 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs" Nov 03 16:53:32.022792 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:32.022765 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs"] Nov 03 16:53:32.024484 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:53:32.024456 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef532337_ef30_48e6_a371_120051ae7ee0.slice/crio-268c338a56618e168d5561035b58f5a973719cd9fa973c306aa8bc0a6296395f WatchSource:0}: Error finding container 268c338a56618e168d5561035b58f5a973719cd9fa973c306aa8bc0a6296395f: Status 404 returned error can't find the container with id 268c338a56618e168d5561035b58f5a973719cd9fa973c306aa8bc0a6296395f Nov 03 16:53:32.403853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:32.403776 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs" event={"ID":"ef532337-ef30-48e6-a371-120051ae7ee0","Type":"ContainerStarted","Data":"392da0a5cd83500c92de07104947547edc342933b7366a45601c71f051fa59de"} Nov 03 16:53:32.403853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:32.403832 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs" event={"ID":"ef532337-ef30-48e6-a371-120051ae7ee0","Type":"ContainerStarted","Data":"268c338a56618e168d5561035b58f5a973719cd9fa973c306aa8bc0a6296395f"} Nov 03 16:53:32.404294 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:32.403876 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs" Nov 03 16:53:32.473839 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:32.467853 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs" podStartSLOduration=1.467833286 podStartE2EDuration="1.467833286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:32.465140542 +0000 UTC m=+2356.054140549" watchObservedRunningTime="2026-11-03 16:53:32.467833286 +0000 UTC m=+2356.056833255" Nov 03 16:53:38.624907 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:38.624864 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:38.960811 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:38.959880 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc"] Nov 03 16:53:38.968730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:38.966767 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Nov 03 16:53:38.977718 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:38.972962 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.024439 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.019734 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" podUID="2f4b9526-8df4-4d0f-a9bc-34824126f4ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:39.027738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.025744 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc"] Nov 03 16:53:39.053722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.049491 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.061836 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.061802 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.062096 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.062073 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ng84\" (UniqueName: \"kubernetes.io/projected/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-kube-api-access-2ng84\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.062246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.062230 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.115892 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.115847 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.128963 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.116123 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.140445 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.129271 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.140844 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.140808 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.163841 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.163808 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.164219 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.058926 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:53:39.265935 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.265903 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.265935 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.265941 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.266198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.265962 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ng84\" (UniqueName: \"kubernetes.io/projected/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-kube-api-access-2ng84\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.266198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.265998 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.266198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.266053 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.266198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.266075 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.266198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.266118 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.266198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.266171 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.266493 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.266206 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.266493 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.266331 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.266618 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.266591 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.266776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.266667 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.266891 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.266870 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.266978 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.266962 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.268980 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.268957 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.269075 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.269005 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.290800 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.290776 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ng84\" (UniqueName: \"kubernetes.io/projected/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-kube-api-access-2ng84\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.290905 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.290819 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.367623 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.367597 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:39.687926 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:53:39.687900 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb55a71_37cf_4f12_bb4e_6edef8a3db2e.slice/crio-b4bb20628f0f54a74346e5eeafe5c0ad52dd66768df2a8f86a3bbc635c5bf271 WatchSource:0}: Error finding container b4bb20628f0f54a74346e5eeafe5c0ad52dd66768df2a8f86a3bbc635c5bf271: Status 404 returned error can't find the container with id b4bb20628f0f54a74346e5eeafe5c0ad52dd66768df2a8f86a3bbc635c5bf271 Nov 03 16:53:39.688382 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.688358 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc"] Nov 03 16:53:39.690515 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.690485 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:53:39.690604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.690553 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:53:39.690604 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:39.690578 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:53:40.504834 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:40.504441 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" event={"ID":"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e","Type":"ContainerStarted","Data":"52aff2feb0520554e0777b6ae637b85f0b2abea2e89aa85b6ea4d5c2329a2384"} Nov 03 16:53:40.505077 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:40.504941 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" event={"ID":"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e","Type":"ContainerStarted","Data":"b4bb20628f0f54a74346e5eeafe5c0ad52dd66768df2a8f86a3bbc635c5bf271"} Nov 03 16:53:40.572109 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:40.572060 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" podStartSLOduration=2.57204507 podStartE2EDuration="2.57204507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:40.569801864 +0000 UTC m=+2364.158801831" watchObservedRunningTime="2026-11-03 16:53:40.57204507 +0000 UTC m=+2364.161045036" Nov 03 16:53:41.368402 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:41.368371 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:41.370153 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:41.370131 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 16:53:41.370218 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:41.370181 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" podUID="1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 16:53:42.368216 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:42.368183 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 16:53:42.368416 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:42.368243 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" podUID="1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 16:53:43.368114 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:43.368078 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 16:53:43.368289 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:43.368143 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" podUID="1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 16:53:43.424522 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:43.424495 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs" Nov 03 16:53:44.371465 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:44.370131 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 16:53:44.371465 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:44.370195 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" podUID="1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 16:53:45.368817 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:45.368780 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 16:53:45.369196 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:45.368841 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" podUID="1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 16:53:46.368821 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:46.368779 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 16:53:46.369215 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:46.368850 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" podUID="1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 16:53:47.373520 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:47.373491 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:47.373960 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:47.373884 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:47.374621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:47.374603 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 16:53:47.436363 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:47.436337 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw"] Nov 03 16:53:47.436565 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:47.436542 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" podUID="6db8e691-22a1-48fc-9edc-2454692a72f0" containerName="istio-proxy" containerID="cri-o://f2cac98e47b7c2f8fc1649a41dcb68cf124bff1b04a0321a990068e3eafef93e" gracePeriod=30 Nov 03 16:53:52.718062 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.717965 4947 generic.go:356] "Generic (PLEG): container finished" podID="6db8e691-22a1-48fc-9edc-2454692a72f0" containerID="f2cac98e47b7c2f8fc1649a41dcb68cf124bff1b04a0321a990068e3eafef93e" exitCode=0 Nov 03 16:53:52.718985 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.718033 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" event={"ID":"6db8e691-22a1-48fc-9edc-2454692a72f0","Type":"ContainerDied","Data":"f2cac98e47b7c2f8fc1649a41dcb68cf124bff1b04a0321a990068e3eafef93e"} Nov 03 16:53:52.749973 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.749953 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:53:52.781640 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.781621 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db8e691-22a1-48fc-9edc-2454692a72f0-kube-api-access-wr2md\" (UniqueName: \"kubernetes.io/projected/6db8e691-22a1-48fc-9edc-2454692a72f0-kube-api-access-wr2md\") pod \"6db8e691-22a1-48fc-9edc-2454692a72f0\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " Nov 03 16:53:52.781784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.781686 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-token\" (UniqueName: \"kubernetes.io/projected/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-token\") pod \"6db8e691-22a1-48fc-9edc-2454692a72f0\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " Nov 03 16:53:52.781784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.781751 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-envoy\") pod \"6db8e691-22a1-48fc-9edc-2454692a72f0\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " Nov 03 16:53:52.781784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.781774 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db8e691-22a1-48fc-9edc-2454692a72f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db8e691-22a1-48fc-9edc-2454692a72f0-istiod-ca-cert\") pod \"6db8e691-22a1-48fc-9edc-2454692a72f0\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " Nov 03 16:53:52.781941 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.781792 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-data\") pod \"6db8e691-22a1-48fc-9edc-2454692a72f0\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " Nov 03 16:53:52.781941 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.781818 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-workload-socket\") pod \"6db8e691-22a1-48fc-9edc-2454692a72f0\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " Nov 03 16:53:52.781941 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.781843 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-credential-socket\") pod \"6db8e691-22a1-48fc-9edc-2454692a72f0\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " Nov 03 16:53:52.782096 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.782025 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-podinfo\") pod \"6db8e691-22a1-48fc-9edc-2454692a72f0\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " Nov 03 16:53:52.782158 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.782107 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-workload-certs\") pod \"6db8e691-22a1-48fc-9edc-2454692a72f0\" (UID: \"6db8e691-22a1-48fc-9edc-2454692a72f0\") " Nov 03 16:53:52.782215 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.782147 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-workload-socket" pod "6db8e691-22a1-48fc-9edc-2454692a72f0" (UID: "6db8e691-22a1-48fc-9edc-2454692a72f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:53:52.782270 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.782229 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db8e691-22a1-48fc-9edc-2454692a72f0-istiod-ca-cert" pod "6db8e691-22a1-48fc-9edc-2454692a72f0" (UID: "6db8e691-22a1-48fc-9edc-2454692a72f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:53:52.782270 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.782242 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-data" pod "6db8e691-22a1-48fc-9edc-2454692a72f0" (UID: "6db8e691-22a1-48fc-9edc-2454692a72f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:53:52.782480 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.782459 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-credential-socket" pod "6db8e691-22a1-48fc-9edc-2454692a72f0" (UID: "6db8e691-22a1-48fc-9edc-2454692a72f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:53:52.782640 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.782608 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-workload-certs" pod "6db8e691-22a1-48fc-9edc-2454692a72f0" (UID: "6db8e691-22a1-48fc-9edc-2454692a72f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:53:52.782849 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.782829 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6db8e691-22a1-48fc-9edc-2454692a72f0-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:53:52.782928 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.782855 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:53:52.782928 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.782869 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:53:52.783049 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.782922 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:53:52.783049 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.783007 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:53:52.784027 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.783998 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db8e691-22a1-48fc-9edc-2454692a72f0-kube-api-access-wr2md" pod "6db8e691-22a1-48fc-9edc-2454692a72f0" (UID: "6db8e691-22a1-48fc-9edc-2454692a72f0"). InnerVolumeSpecName "kube-api-access-wr2md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:52.784131 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.784030 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-envoy" pod "6db8e691-22a1-48fc-9edc-2454692a72f0" (UID: "6db8e691-22a1-48fc-9edc-2454692a72f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:53:52.784182 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.784157 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-token" pod "6db8e691-22a1-48fc-9edc-2454692a72f0" (UID: "6db8e691-22a1-48fc-9edc-2454692a72f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:53:52.784402 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.784378 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-podinfo" pod "6db8e691-22a1-48fc-9edc-2454692a72f0" (UID: "6db8e691-22a1-48fc-9edc-2454692a72f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:53:52.883982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.883953 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr2md\" (UniqueName: \"kubernetes.io/projected/6db8e691-22a1-48fc-9edc-2454692a72f0-kube-api-access-wr2md\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:53:52.883982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.883977 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:53:52.884172 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.883993 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:53:52.884172 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:52.884001 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6db8e691-22a1-48fc-9edc-2454692a72f0-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:53:53.726198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:53.726165 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" Nov 03 16:53:53.726198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:53.726181 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw" event={"ID":"6db8e691-22a1-48fc-9edc-2454692a72f0","Type":"ContainerDied","Data":"a114746012a81f93150ee2f0b9b8660de266f3ee6930589876487a7cc6f76943"} Nov 03 16:53:53.726825 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:53.726248 4947 scope.go:122] "RemoveContainer" containerID="f2cac98e47b7c2f8fc1649a41dcb68cf124bff1b04a0321a990068e3eafef93e" Nov 03 16:53:53.798514 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:53.798487 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw"] Nov 03 16:53:53.812059 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:53.812035 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-sr2vw"] Nov 03 16:53:54.008473 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.008441 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj"] Nov 03 16:53:54.009362 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.009340 4947 cpu_manager.go:414] "containerMap: removing container" podUID="6db8e691-22a1-48fc-9edc-2454692a72f0" containerName="istio-proxy" Nov 03 16:53:54.009468 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.009364 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db8e691-22a1-48fc-9edc-2454692a72f0" containerName="istio-proxy" Nov 03 16:53:54.009520 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.009507 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db8e691-22a1-48fc-9edc-2454692a72f0" containerName="istio-proxy" Nov 03 16:53:54.014679 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.014658 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.044623 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.044594 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj"] Nov 03 16:53:54.095691 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.095658 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.095846 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.095822 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.096091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.095924 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.096091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.096051 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.096319 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.096111 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.096319 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.096188 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.096319 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.096248 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vx6j\" (UniqueName: \"kubernetes.io/projected/cf3d6e3e-c739-4f9f-982a-16856d80e08c-kube-api-access-8vx6j\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.096431 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.096356 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.096431 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.096378 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.197969 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.197939 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.198143 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.198014 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.198143 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.198082 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.198261 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.198180 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.198261 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.198243 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.198369 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.198275 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.198369 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.198319 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-data\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.198369 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.198322 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vx6j\" (UniqueName: \"kubernetes.io/projected/cf3d6e3e-c739-4f9f-982a-16856d80e08c-kube-api-access-8vx6j\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.198543 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.198405 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-credential-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.198543 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.198446 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.198543 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.198485 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-workload-certs\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.198543 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.198487 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.198784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.198745 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-workload-socket\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.199008 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.198937 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istiod-ca-cert\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.200502 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.200479 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-envoy\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.200603 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.200550 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-podinfo\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.223029 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.223005 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-token\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.223215 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.223198 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vx6j\" (UniqueName: \"kubernetes.io/projected/cf3d6e3e-c739-4f9f-982a-16856d80e08c-kube-api-access-8vx6j\") pod \"e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.334209 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.334139 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:54.564807 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.564774 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db8e691-22a1-48fc-9edc-2454692a72f0" path="/var/lib/kubelet/pods/6db8e691-22a1-48fc-9edc-2454692a72f0/volumes" Nov 03 16:53:54.710279 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.710249 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj"] Nov 03 16:53:54.710634 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:53:54.710611 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3d6e3e_c739_4f9f_982a_16856d80e08c.slice/crio-ee38db02d29b17ebad394cf10ce144be7b4308365218c324528db0e656bb42bb WatchSource:0}: Error finding container ee38db02d29b17ebad394cf10ce144be7b4308365218c324528db0e656bb42bb: Status 404 returned error can't find the container with id ee38db02d29b17ebad394cf10ce144be7b4308365218c324528db0e656bb42bb Nov 03 16:53:54.713159 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.713128 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:53:54.713232 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.713194 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:53:54.713232 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.713224 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:53:54.737792 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:54.737762 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" event={"ID":"cf3d6e3e-c739-4f9f-982a-16856d80e08c","Type":"ContainerStarted","Data":"ee38db02d29b17ebad394cf10ce144be7b4308365218c324528db0e656bb42bb"} Nov 03 16:53:55.745691 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:55.745660 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" event={"ID":"cf3d6e3e-c739-4f9f-982a-16856d80e08c","Type":"ContainerStarted","Data":"f402416917c0d17b1a7a2d8fe210291d129a2f007742bf708b4cc366288aee16"} Nov 03 16:53:55.824389 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:55.824312 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" podStartSLOduration=2.824292632 podStartE2EDuration="2.824292632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:53:55.81018901 +0000 UTC m=+2379.399189002" watchObservedRunningTime="2026-11-03 16:53:55.824292632 +0000 UTC m=+2379.413292602" Nov 03 16:53:56.334566 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:56.334532 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:53:56.335857 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:56.335832 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 16:53:56.336001 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:56.335881 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" podUID="cf3d6e3e-c739-4f9f-982a-16856d80e08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 16:53:57.334684 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:57.334607 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 16:53:57.334684 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:57.334660 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" podUID="cf3d6e3e-c739-4f9f-982a-16856d80e08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 16:53:57.690405 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:57.690330 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:53:58.051619 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.051584 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42"] Nov 03 16:53:58.056668 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.056645 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.083770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.083735 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42"] Nov 03 16:53:58.090659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.090625 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-3dd067-openshift-default-dockercfg-6m45q\"" Nov 03 16:53:58.097533 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.097505 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-envoy\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.097688 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.097552 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-credential-socket\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.097688 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.097582 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-token\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.097688 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.097644 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-workload-certs\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.097688 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.097674 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9c529d-c795-41ac-8145-e126598ecc4e-istiod-ca-cert\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.121032 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.097737 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-data\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.121032 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.097783 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cwx9\" (UniqueName: \"kubernetes.io/projected/2b9c529d-c795-41ac-8145-e126598ecc4e-kube-api-access-5cwx9\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.121032 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.097812 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-workload-socket\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.121032 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.097855 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-podinfo\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.200602 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.200561 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cwx9\" (UniqueName: \"kubernetes.io/projected/2b9c529d-c795-41ac-8145-e126598ecc4e-kube-api-access-5cwx9\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.200922 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.200893 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-workload-socket\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.201116 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.201099 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-podinfo\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.201429 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.201408 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-envoy\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.201569 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.201553 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-credential-socket\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.205754 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.205731 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-workload-socket\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.205897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.205759 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-token\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.206100 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.206079 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-workload-certs\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.209786 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.209761 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9c529d-c795-41ac-8145-e126598ecc4e-istiod-ca-cert\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.233720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.228854 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-data\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.233720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.229396 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-data\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.233720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.229924 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-workload-certs\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.233720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.230426 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-credential-socket\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.233720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.231045 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9c529d-c795-41ac-8145-e126598ecc4e-istiod-ca-cert\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.234132 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.233934 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-envoy\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.264994 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.244217 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-podinfo\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.282373 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.280258 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-token\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.290549 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.290476 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cwx9\" (UniqueName: \"kubernetes.io/projected/2b9c529d-c795-41ac-8145-e126598ecc4e-kube-api-access-5cwx9\") pod \"e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.335326 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.334417 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 16:53:58.335326 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.334523 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" podUID="cf3d6e3e-c739-4f9f-982a-16856d80e08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 16:53:58.383342 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.383231 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:53:58.766756 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.750007 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:53:58.766756 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.750108 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:53:58.766756 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.750181 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:53:58.766756 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.762619 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42"] Nov 03 16:53:58.899302 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:58.899250 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" event={"ID":"2b9c529d-c795-41ac-8145-e126598ecc4e","Type":"ContainerStarted","Data":"7791b0c147efb8a0ac153c8400b90fb93e95aaacbb60f1a8cf467e3e61b225b3"} Nov 03 16:53:59.411773 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:59.399625 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 16:53:59.411773 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:59.399721 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" podUID="cf3d6e3e-c739-4f9f-982a-16856d80e08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 16:53:59.971723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:53:59.968854 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" event={"ID":"2b9c529d-c795-41ac-8145-e126598ecc4e","Type":"ContainerStarted","Data":"038c2cadb91ff28ef9731c3ea519ce7bf9f90a73dc2da6fa18d29f4e484c933b"} Nov 03 16:54:00.349481 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:00.349061 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 16:54:00.349481 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:00.349122 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" podUID="cf3d6e3e-c739-4f9f-982a-16856d80e08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 16:54:00.387129 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:00.387089 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:54:00.424008 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:00.423975 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:54:00.473904 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:54:00.473869 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:54:00.478468 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:00.478417 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" podStartSLOduration=2.478398694 podStartE2EDuration="2.478398694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:00.075055186 +0000 UTC m=+2383.664055245" watchObservedRunningTime="2026-11-03 16:54:00.478398694 +0000 UTC m=+2384.067398663" Nov 03 16:54:00.996147 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:00.995414 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded" Nov 03 16:54:00.998062 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:00.998041 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:54:01.010986 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:01.010955 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:54:01.014804 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:01.014764 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" podUID="2f4b9526-8df4-4d0f-a9bc-34824126f4ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:01.069730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:01.065836 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:01.298774 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:01.292906 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:01.298774 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:01.293352 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 16:54:01.356745 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:01.341484 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:01.378934 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:01.378885 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 16:54:01.379239 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:01.379193 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" podUID="cf3d6e3e-c739-4f9f-982a-16856d80e08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 16:54:01.619886 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:01.619783 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:01.984786 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:01.983784 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:02.021718 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.019196 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:02.021718 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.020651 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:54:02.031726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.026764 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" podUID="2f4b9526-8df4-4d0f-a9bc-34824126f4ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:02.031726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.028043 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:54:02.067850 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.067811 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:02.090722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.073270 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:54:02.381711 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.378502 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:02.381711 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.378996 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 16:54:02.381711 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.379188 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 16:54:02.381711 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.379717 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:54:02.381711 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.379777 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:54:02.381711 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.379813 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:54:02.459722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.438996 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 16:54:02.459722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.449772 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" podUID="cf3d6e3e-c739-4f9f-982a-16856d80e08c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 16:54:02.630724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.627914 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded" Nov 03 16:54:02.731365 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.731286 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:54:02.731623 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.731607 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 16:54:02.782957 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.782926 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 16:54:02.884331 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.884299 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 16:54:02.918426 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.918400 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 16:54:02.918684 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.918670 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 16:54:02.958986 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.958961 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 16:54:02.960267 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:02.960167 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 16:54:03.237767 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.237734 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq"] Nov 03 16:54:03.243751 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.243690 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.258066 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.258035 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mj2m\" (UniqueName: \"kubernetes.io/projected/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-kube-api-access-7mj2m\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.258840 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.258818 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.259331 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.259306 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.259512 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.259493 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.264215 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.264194 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.264733 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.264691 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.265287 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.265162 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.265460 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.265440 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.265901 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.265870 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.271209 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.271138 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq"] Nov 03 16:54:03.276068 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.276049 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8"] Nov 03 16:54:03.284835 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.284809 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.327476 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.327135 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8"] Nov 03 16:54:03.367118 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.367091 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.367667 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.367585 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.367867 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.367842 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wb5r\" (UniqueName: \"kubernetes.io/projected/541d8697-b5fa-467e-878c-848106509998-kube-api-access-8wb5r\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.367969 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.367958 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.368065 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.368056 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.368158 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.368147 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/541d8697-b5fa-467e-878c-848106509998-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.368236 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.368227 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mj2m\" (UniqueName: \"kubernetes.io/projected/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-kube-api-access-7mj2m\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.368303 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.368295 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.368366 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.368359 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/541d8697-b5fa-467e-878c-848106509998-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.368438 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.368430 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.368509 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.368499 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.368753 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.368727 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.368822 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.368790 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.368854 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.368824 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.368889 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.368853 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.368889 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.368881 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/541d8697-b5fa-467e-878c-848106509998-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.369053 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.368950 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.369053 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.368974 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.369053 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.369007 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.369229 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.369211 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.369786 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.369648 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.370653 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.370137 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.370653 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.370204 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.373236 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.373187 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.375110 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.375089 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.417631 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.417590 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.422846 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.422814 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mj2m\" (UniqueName: \"kubernetes.io/projected/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-kube-api-access-7mj2m\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.458337 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.457669 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:54:03.458740 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.458676 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:54:03.459630 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.459608 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 16:54:03.470282 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.470250 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/541d8697-b5fa-467e-878c-848106509998-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.470418 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.470304 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.470489 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.470420 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.470489 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.470451 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.470489 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.470484 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/541d8697-b5fa-467e-878c-848106509998-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.470648 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.470563 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wb5r\" (UniqueName: \"kubernetes.io/projected/541d8697-b5fa-467e-878c-848106509998-kube-api-access-8wb5r\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.470648 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.470597 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.470786 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.470656 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.470786 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.470736 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/541d8697-b5fa-467e-878c-848106509998-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.470937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.470917 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-workload-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.471191 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.471166 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-credential-socket\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.471680 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.471655 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-istio-data\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.471792 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.471675 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-workload-certs\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.471874 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.471851 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/541d8697-b5fa-467e-878c-848106509998-istiod-ca-cert\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.473432 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.473405 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-istio-envoy\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.474359 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.474333 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/541d8697-b5fa-467e-878c-848106509998-istio-podinfo\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.494844 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.494816 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/541d8697-b5fa-467e-878c-848106509998-istio-token\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.495464 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.495446 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wb5r\" (UniqueName: \"kubernetes.io/projected/541d8697-b5fa-467e-878c-848106509998-kube-api-access-8wb5r\") pod \"e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.562596 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.562560 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:03.608848 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.608815 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:03.899579 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.897264 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq"] Nov 03 16:54:03.899579 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:54:03.898117 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a6b2fc_a875_428c_a726_bfd9fc2aaa85.slice/crio-28e32dd5e5b19510d8079303abcbda944fe9c8c2340decf4a430094b544f6366 WatchSource:0}: Error finding container 28e32dd5e5b19510d8079303abcbda944fe9c8c2340decf4a430094b544f6366: Status 404 returned error can't find the container with id 28e32dd5e5b19510d8079303abcbda944fe9c8c2340decf4a430094b544f6366 Nov 03 16:54:03.901187 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.901152 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:54:03.901270 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.901218 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:54:03.901270 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.901245 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:54:03.943455 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:54:03.943429 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541d8697_b5fa_467e_878c_848106509998.slice/crio-f162f847eb7085183c61f3e3d78e03f5d2dac0f1215485019a1708ad13131ef3 WatchSource:0}: Error finding container f162f847eb7085183c61f3e3d78e03f5d2dac0f1215485019a1708ad13131ef3: Status 404 returned error can't find the container with id f162f847eb7085183c61f3e3d78e03f5d2dac0f1215485019a1708ad13131ef3 Nov 03 16:54:03.946002 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.945976 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8"] Nov 03 16:54:03.946119 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.946088 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:54:03.946167 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.946153 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:54:03.946200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:03.946180 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:54:04.170421 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.170315 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" event={"ID":"80a6b2fc-a875-428c-a726-bfd9fc2aaa85","Type":"ContainerStarted","Data":"75df66069fec869f5fed604bc7d99137a4fcde1214ed51d20f052a1381fba32d"} Nov 03 16:54:04.170793 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.170745 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" event={"ID":"80a6b2fc-a875-428c-a726-bfd9fc2aaa85","Type":"ContainerStarted","Data":"28e32dd5e5b19510d8079303abcbda944fe9c8c2340decf4a430094b544f6366"} Nov 03 16:54:04.176859 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.176469 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" event={"ID":"541d8697-b5fa-467e-878c-848106509998","Type":"ContainerStarted","Data":"de8ccf365c3c707050fd8c976e3fb5c7edf0062b1b6c0a96aa74f2243ddee869"} Nov 03 16:54:04.176859 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.176519 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" event={"ID":"541d8697-b5fa-467e-878c-848106509998","Type":"ContainerStarted","Data":"f162f847eb7085183c61f3e3d78e03f5d2dac0f1215485019a1708ad13131ef3"} Nov 03 16:54:04.182628 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.182599 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55"] Nov 03 16:54:04.280852 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.261906 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.280852 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.280459 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" podStartSLOduration=1.28042054 podStartE2EDuration="1.28042054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:04.259396229 +0000 UTC m=+2387.848396198" watchObservedRunningTime="2026-11-03 16:54:04.28042054 +0000 UTC m=+2387.869420511" Nov 03 16:54:04.281194 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.281172 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55"] Nov 03 16:54:04.348825 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.331186 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" podStartSLOduration=1.331166708 podStartE2EDuration="1.331166708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:04.329260049 +0000 UTC m=+2387.918260018" watchObservedRunningTime="2026-11-03 16:54:04.331166708 +0000 UTC m=+2387.920166677" Nov 03 16:54:04.397200 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.397154 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.397396 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.397214 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.397491 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.397394 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.397616 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.397595 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.409503 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.397855 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q47xs\" (UniqueName: \"kubernetes.io/projected/d2a9d92e-90e9-41b4-9eba-1faf4b485843-kube-api-access-q47xs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.409503 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.397927 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.409503 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.397971 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.409503 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.398030 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.409503 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.398067 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.499392 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.499352 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q47xs\" (UniqueName: \"kubernetes.io/projected/d2a9d92e-90e9-41b4-9eba-1faf4b485843-kube-api-access-q47xs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.499582 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.499447 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.499582 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.499495 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.499582 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.499557 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.499813 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.499592 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.499813 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.499627 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.499813 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.499670 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.499813 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.499717 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.499813 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.499767 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.501509 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.501481 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-credential-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.501630 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.501618 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-data\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.501760 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.501730 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-workload-socket\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.501931 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.501891 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-workload-certs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.502487 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.502458 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istiod-ca-cert\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.503723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.503659 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-podinfo\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.519434 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.519402 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-envoy\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.563987 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.562338 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-token\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.563987 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.562462 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q47xs\" (UniqueName: \"kubernetes.io/projected/d2a9d92e-90e9-41b4-9eba-1faf4b485843-kube-api-access-q47xs\") pod \"e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:04.574852 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.574817 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 16:54:04.575094 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.574874 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" podUID="80a6b2fc-a875-428c-a726-bfd9fc2aaa85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 16:54:04.637340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.637292 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:04.637340 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.637350 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:04.765783 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:04.762915 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:05.564842 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:05.563117 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 16:54:05.564842 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:05.563178 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" podUID="80a6b2fc-a875-428c-a726-bfd9fc2aaa85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 16:54:05.650154 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:05.650109 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 16:54:05.650342 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:05.650174 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" podUID="541d8697-b5fa-467e-878c-848106509998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 16:54:05.748781 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:05.740773 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55"] Nov 03 16:54:05.761720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:05.761301 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:54:05.761720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:05.761396 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:54:05.761720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:05.761440 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:54:06.396473 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:06.396436 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" event={"ID":"d2a9d92e-90e9-41b4-9eba-1faf4b485843","Type":"ContainerStarted","Data":"1dc605142923274967868572266f044c86a88992288bf7c87039f36d09a67bb6"} Nov 03 16:54:06.396661 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:06.396514 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" event={"ID":"d2a9d92e-90e9-41b4-9eba-1faf4b485843","Type":"ContainerStarted","Data":"59d60effde11c315e9ba33c458886a203130bf92be69ddff4bed11b557bfd229"} Nov 03 16:54:06.481436 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:06.481370 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" podStartSLOduration=2.481352741 podStartE2EDuration="2.481352741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:06.476976021 +0000 UTC m=+2390.065975989" watchObservedRunningTime="2026-11-03 16:54:06.481352741 +0000 UTC m=+2390.070352710" Nov 03 16:54:06.563346 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:06.563301 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 16:54:06.563556 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:06.563366 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" podUID="80a6b2fc-a875-428c-a726-bfd9fc2aaa85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 16:54:06.622327 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:06.622286 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 16:54:06.622932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:06.622898 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" podUID="541d8697-b5fa-467e-878c-848106509998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 16:54:06.765889 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:06.765780 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:06.766256 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:06.766229 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 16:54:06.766474 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:06.766440 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" podUID="d2a9d92e-90e9-41b4-9eba-1faf4b485843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 16:54:07.599724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:07.586270 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 16:54:07.599724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:07.586326 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" podUID="80a6b2fc-a875-428c-a726-bfd9fc2aaa85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 16:54:07.692902 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:07.691306 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 16:54:07.692902 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:07.691368 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" podUID="541d8697-b5fa-467e-878c-848106509998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 16:54:07.768723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:07.768044 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 16:54:07.768723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:07.768104 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" podUID="d2a9d92e-90e9-41b4-9eba-1faf4b485843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 16:54:08.589721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:08.573008 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 16:54:08.589721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:08.583799 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" podUID="80a6b2fc-a875-428c-a726-bfd9fc2aaa85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 16:54:08.619722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:08.617957 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 16:54:08.619722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:08.618014 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" podUID="541d8697-b5fa-467e-878c-848106509998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 16:54:08.777725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:08.766040 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 16:54:08.777725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:08.766098 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" podUID="d2a9d92e-90e9-41b4-9eba-1faf4b485843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 16:54:09.577332 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:09.577297 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 16:54:09.577621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:09.577591 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" podUID="80a6b2fc-a875-428c-a726-bfd9fc2aaa85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 16:54:09.621721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:09.618075 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 16:54:09.621721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:09.618154 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" podUID="541d8697-b5fa-467e-878c-848106509998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 16:54:09.778725 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:09.767465 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 16:54:09.784729 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:09.779310 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" podUID="d2a9d92e-90e9-41b4-9eba-1faf4b485843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 16:54:10.582722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:10.580786 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 16:54:10.582722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:10.580856 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" podUID="80a6b2fc-a875-428c-a726-bfd9fc2aaa85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 16:54:10.623719 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:10.623572 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 16:54:10.623719 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:10.623645 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" podUID="541d8697-b5fa-467e-878c-848106509998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 16:54:10.764873 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:10.764835 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 16:54:10.765045 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:10.764898 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" podUID="d2a9d92e-90e9-41b4-9eba-1faf4b485843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 16:54:11.565877 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:11.565837 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" start-of-body= Nov 03 16:54:11.568243 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:11.568164 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" podUID="80a6b2fc-a875-428c-a726-bfd9fc2aaa85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.146:15021/healthz/ready\": dial tcp 10.132.0.146:15021: connect: connection refused" Nov 03 16:54:11.616723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:11.612197 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 16:54:11.616723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:11.612251 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" podUID="541d8697-b5fa-467e-878c-848106509998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 16:54:11.765205 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:11.765166 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 16:54:11.765414 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:11.765242 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" podUID="d2a9d92e-90e9-41b4-9eba-1faf4b485843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 16:54:12.614378 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:12.614346 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:12.614893 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:12.614615 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:12.618580 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:12.618540 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:54:12.661076 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:12.661046 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:12.661750 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:12.661729 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:12.663339 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:12.663295 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:54:12.764267 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:12.764135 4947 patch_prober.go:29] interesting pod/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 16:54:12.764267 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:12.764189 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" podUID="d2a9d92e-90e9-41b4-9eba-1faf4b485843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 16:54:13.814066 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:13.814034 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:13.814657 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:13.814462 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:13.815248 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:13.815231 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 16:54:16.891225 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:16.891202 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:54:16.892614 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:16.892590 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:54:17.446730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:17.420900 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7"] Nov 03 16:54:17.449490 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:17.449454 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7"] Nov 03 16:54:17.449833 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:17.449811 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" Nov 03 16:54:17.478283 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:17.475594 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-3b6f35\"" Nov 03 16:54:17.478283 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:17.475768 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-3b6f35-dockercfg-g6lrb\"" Nov 03 16:54:17.491523 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:17.491459 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftgbf\" (UniqueName: \"kubernetes.io/projected/95d28dd4-0aae-4f71-9bbb-24e102b03b75-kube-api-access-ftgbf\") pod \"payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7\" (UID: \"95d28dd4-0aae-4f71-9bbb-24e102b03b75\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" Nov 03 16:54:17.492785 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:17.492744 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d28dd4-0aae-4f71-9bbb-24e102b03b75-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7\" (UID: \"95d28dd4-0aae-4f71-9bbb-24e102b03b75\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" Nov 03 16:54:17.594342 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:17.594302 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d28dd4-0aae-4f71-9bbb-24e102b03b75-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7\" (UID: \"95d28dd4-0aae-4f71-9bbb-24e102b03b75\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" Nov 03 16:54:17.594559 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:17.594538 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftgbf\" (UniqueName: \"kubernetes.io/projected/95d28dd4-0aae-4f71-9bbb-24e102b03b75-kube-api-access-ftgbf\") pod \"payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7\" (UID: \"95d28dd4-0aae-4f71-9bbb-24e102b03b75\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" Nov 03 16:54:17.595064 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:17.595036 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d28dd4-0aae-4f71-9bbb-24e102b03b75-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7\" (UID: \"95d28dd4-0aae-4f71-9bbb-24e102b03b75\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" Nov 03 16:54:17.619299 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:17.619268 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftgbf\" (UniqueName: \"kubernetes.io/projected/95d28dd4-0aae-4f71-9bbb-24e102b03b75-kube-api-access-ftgbf\") pod \"payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7\" (UID: \"95d28dd4-0aae-4f71-9bbb-24e102b03b75\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" Nov 03 16:54:17.620900 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:17.620878 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m"] Nov 03 16:54:17.656315 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:17.656278 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx"] Nov 03 16:54:17.796908 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:17.796877 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" Nov 03 16:54:18.104484 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:54:18.104443 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d28dd4_0aae_4f71_9bbb_24e102b03b75.slice/crio-72162371a88788fbd415584ed68dbf3b218079315b52efd1094d1e25c247c349 WatchSource:0}: Error finding container 72162371a88788fbd415584ed68dbf3b218079315b52efd1094d1e25c247c349: Status 404 returned error can't find the container with id 72162371a88788fbd415584ed68dbf3b218079315b52efd1094d1e25c247c349 Nov 03 16:54:18.105314 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.105197 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7"] Nov 03 16:54:18.740925 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.740887 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck"] Nov 03 16:54:18.745022 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.745003 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" Nov 03 16:54:18.764601 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.764169 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-3dd067-serving-cert\"" Nov 03 16:54:18.783012 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.769207 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck"] Nov 03 16:54:18.830752 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.830724 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck\" (UID: \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" Nov 03 16:54:18.830922 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.830900 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hs98\" (UniqueName: \"kubernetes.io/projected/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-kube-api-access-2hs98\") pod \"maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck\" (UID: \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" Nov 03 16:54:18.830986 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.830968 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck\" (UID: \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" Nov 03 16:54:18.831118 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.831100 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck\" (UID: \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" Nov 03 16:54:18.834002 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.833969 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" event={"ID":"95d28dd4-0aae-4f71-9bbb-24e102b03b75","Type":"ContainerStarted","Data":"a83593cc161f0eb6f9b58d80bf9d0b9a6bd8f90edfd86517ade85c80badc1c67"} Nov 03 16:54:18.834123 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.834017 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" event={"ID":"95d28dd4-0aae-4f71-9bbb-24e102b03b75","Type":"ContainerStarted","Data":"72162371a88788fbd415584ed68dbf3b218079315b52efd1094d1e25c247c349"} Nov 03 16:54:18.834123 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.834052 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" Nov 03 16:54:18.883042 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.882994 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" podStartSLOduration=1.882981103 podStartE2EDuration="1.882981103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:18.880317268 +0000 UTC m=+2402.469317235" watchObservedRunningTime="2026-11-03 16:54:18.882981103 +0000 UTC m=+2402.471981075" Nov 03 16:54:18.932049 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.932015 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck\" (UID: \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" Nov 03 16:54:18.932207 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.932160 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck\" (UID: \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" Nov 03 16:54:18.932333 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.932314 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hs98\" (UniqueName: \"kubernetes.io/projected/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-kube-api-access-2hs98\") pod \"maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck\" (UID: \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" Nov 03 16:54:18.932397 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.932379 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck\" (UID: \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" Nov 03 16:54:18.932838 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.932815 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck\" (UID: \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" Nov 03 16:54:18.934926 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.934898 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck\" (UID: \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" Nov 03 16:54:18.935050 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.934925 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck\" (UID: \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" Nov 03 16:54:18.960409 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:18.960382 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hs98\" (UniqueName: \"kubernetes.io/projected/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-kube-api-access-2hs98\") pod \"maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck\" (UID: \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" Nov 03 16:54:19.064342 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:19.063988 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" Nov 03 16:54:19.412375 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:54:19.412342 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd78f22e_864e_4ad2_81bb_4cf0589c3d5f.slice/crio-70a90f95fff5467e8093536ed1d8c301e06fc8354ae095a9b663fe2de336cecf WatchSource:0}: Error finding container 70a90f95fff5467e8093536ed1d8c301e06fc8354ae095a9b663fe2de336cecf: Status 404 returned error can't find the container with id 70a90f95fff5467e8093536ed1d8c301e06fc8354ae095a9b663fe2de336cecf Nov 03 16:54:19.412747 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:19.412635 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck"] Nov 03 16:54:19.414005 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:19.413986 4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:54:19.844397 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:19.844350 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" event={"ID":"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f","Type":"ContainerStarted","Data":"6baaf7f918673168b39ebf283326ce23d5128d4823d87e82c05dd8619623f8a3"} Nov 03 16:54:19.844562 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:19.844453 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" event={"ID":"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f","Type":"ContainerStarted","Data":"70a90f95fff5467e8093536ed1d8c301e06fc8354ae095a9b663fe2de336cecf"} Nov 03 16:54:19.844670 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:19.844652 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" Nov 03 16:54:19.950407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:19.949784 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" podStartSLOduration=1.634794344 podStartE2EDuration="1.949765147s" totalImagesPullingTime="314.970803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:19.932146955 +0000 UTC m=+2403.521146925" watchObservedRunningTime="2026-11-03 16:54:19.949765147 +0000 UTC m=+2403.538765117" Nov 03 16:54:20.293930 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:20.293899 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd"] Nov 03 16:54:20.294436 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:20.294405 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" podUID="eb2f23df-d970-4f16-8292-569a9356374c" containerName="maas-api" containerID="cri-o://aea408c87a68e6d37c75b34d9042cd1dcec7cc0ec9a5b1f69c3ce717899cff40" gracePeriod=30 Nov 03 16:54:20.578587 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:20.578548 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" Nov 03 16:54:20.672720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:20.665295 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb2f23df-d970-4f16-8292-569a9356374c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb2f23df-d970-4f16-8292-569a9356374c-openshift-service-ca\") pod \"eb2f23df-d970-4f16-8292-569a9356374c\" (UID: \"eb2f23df-d970-4f16-8292-569a9356374c\") " Nov 03 16:54:20.672720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:20.665363 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb2f23df-d970-4f16-8292-569a9356374c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb2f23df-d970-4f16-8292-569a9356374c-metrics-certs\") pod \"eb2f23df-d970-4f16-8292-569a9356374c\" (UID: \"eb2f23df-d970-4f16-8292-569a9356374c\") " Nov 03 16:54:20.672720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:20.665422 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb2f23df-d970-4f16-8292-569a9356374c-kube-api-access-f9zkn\" (UniqueName: \"kubernetes.io/projected/eb2f23df-d970-4f16-8292-569a9356374c-kube-api-access-f9zkn\") pod \"eb2f23df-d970-4f16-8292-569a9356374c\" (UID: \"eb2f23df-d970-4f16-8292-569a9356374c\") " Nov 03 16:54:20.672720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:20.665591 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb2f23df-d970-4f16-8292-569a9356374c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb2f23df-d970-4f16-8292-569a9356374c-maas-api-tls\") pod \"eb2f23df-d970-4f16-8292-569a9356374c\" (UID: \"eb2f23df-d970-4f16-8292-569a9356374c\") " Nov 03 16:54:20.672720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:20.667019 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb2f23df-d970-4f16-8292-569a9356374c-openshift-service-ca" pod "eb2f23df-d970-4f16-8292-569a9356374c" (UID: "eb2f23df-d970-4f16-8292-569a9356374c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:54:20.699132 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:20.693737 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb2f23df-d970-4f16-8292-569a9356374c-metrics-certs" pod "eb2f23df-d970-4f16-8292-569a9356374c" (UID: "eb2f23df-d970-4f16-8292-569a9356374c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:54:20.712597 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:20.712550 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb2f23df-d970-4f16-8292-569a9356374c-maas-api-tls" pod "eb2f23df-d970-4f16-8292-569a9356374c" (UID: "eb2f23df-d970-4f16-8292-569a9356374c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:54:20.717913 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:20.717867 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb2f23df-d970-4f16-8292-569a9356374c-kube-api-access-f9zkn" pod "eb2f23df-d970-4f16-8292-569a9356374c" (UID: "eb2f23df-d970-4f16-8292-569a9356374c"). InnerVolumeSpecName "kube-api-access-f9zkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:54:20.800046 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:20.799939 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb2f23df-d970-4f16-8292-569a9356374c-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:54:20.812868 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:20.812802 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb2f23df-d970-4f16-8292-569a9356374c-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:54:20.813125 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:20.813110 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9zkn\" (UniqueName: \"kubernetes.io/projected/eb2f23df-d970-4f16-8292-569a9356374c-kube-api-access-f9zkn\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:54:20.823728 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:20.813236 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb2f23df-d970-4f16-8292-569a9356374c-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:54:21.026840 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.026806 4947 generic.go:356] "Generic (PLEG): container finished" podID="eb2f23df-d970-4f16-8292-569a9356374c" containerID="aea408c87a68e6d37c75b34d9042cd1dcec7cc0ec9a5b1f69c3ce717899cff40" exitCode=0 Nov 03 16:54:21.027847 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.027821 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" Nov 03 16:54:21.032139 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.032106 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" event={"ID":"eb2f23df-d970-4f16-8292-569a9356374c","Type":"ContainerDied","Data":"aea408c87a68e6d37c75b34d9042cd1dcec7cc0ec9a5b1f69c3ce717899cff40"} Nov 03 16:54:21.032402 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.032377 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd" event={"ID":"eb2f23df-d970-4f16-8292-569a9356374c","Type":"ContainerDied","Data":"32cefc6acf0de3f890c48108ad53db25eb5319c7f4d33991c53a867a45b5c674"} Nov 03 16:54:21.032556 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.032540 4947 scope.go:122] "RemoveContainer" containerID="aea408c87a68e6d37c75b34d9042cd1dcec7cc0ec9a5b1f69c3ce717899cff40" Nov 03 16:54:21.061646 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.061617 4947 scope.go:122] "RemoveContainer" containerID="aea408c87a68e6d37c75b34d9042cd1dcec7cc0ec9a5b1f69c3ce717899cff40" Nov 03 16:54:21.062327 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:54:21.062302 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea408c87a68e6d37c75b34d9042cd1dcec7cc0ec9a5b1f69c3ce717899cff40\": container with ID starting with aea408c87a68e6d37c75b34d9042cd1dcec7cc0ec9a5b1f69c3ce717899cff40 not found: ID does not exist" containerID="aea408c87a68e6d37c75b34d9042cd1dcec7cc0ec9a5b1f69c3ce717899cff40" Nov 03 16:54:21.062500 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.062470 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea408c87a68e6d37c75b34d9042cd1dcec7cc0ec9a5b1f69c3ce717899cff40"} err="failed to get container status \"aea408c87a68e6d37c75b34d9042cd1dcec7cc0ec9a5b1f69c3ce717899cff40\": rpc error: code = NotFound desc = could not find container \"aea408c87a68e6d37c75b34d9042cd1dcec7cc0ec9a5b1f69c3ce717899cff40\": container with ID starting with aea408c87a68e6d37c75b34d9042cd1dcec7cc0ec9a5b1f69c3ce717899cff40 not found: ID does not exist" Nov 03 16:54:21.140431 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.138509 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd"] Nov 03 16:54:21.146848 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.146809 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-be8495-6fb64c5cc9-vlchd"] Nov 03 16:54:21.526184 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.526149 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr"] Nov 03 16:54:21.527371 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.527346 4947 cpu_manager.go:414] "containerMap: removing container" podUID="eb2f23df-d970-4f16-8292-569a9356374c" containerName="maas-api" Nov 03 16:54:21.527545 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.527529 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb2f23df-d970-4f16-8292-569a9356374c" containerName="maas-api" Nov 03 16:54:21.527872 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.527854 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb2f23df-d970-4f16-8292-569a9356374c" containerName="maas-api" Nov 03 16:54:21.539790 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.539767 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.552198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.552169 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-workload-certs\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.552414 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.552390 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-workload-socket\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.552551 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.552534 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm62r\" (UniqueName: \"kubernetes.io/projected/f6e02969-c94d-47bf-9b80-278e570b6e0c-kube-api-access-cm62r\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.552796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.552780 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6e02969-c94d-47bf-9b80-278e570b6e0c-istiod-ca-cert\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.562834 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.552909 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-data\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.628301 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.628267 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-credential-socket\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.628974 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.628948 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-token\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.629185 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.629165 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-envoy\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.629397 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.629376 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-podinfo\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.629650 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.576512 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-9a80f0-openshift-default-dockercfg-skbfb\"" Nov 03 16:54:21.629819 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.606367 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr"] Nov 03 16:54:21.746721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.733387 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6e02969-c94d-47bf-9b80-278e570b6e0c-istiod-ca-cert\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.746721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.733477 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-data\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.746721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.733528 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-credential-socket\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.746721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.733577 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-token\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.746721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.733642 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-envoy\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.746721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.733732 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-podinfo\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.746721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.733811 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-workload-certs\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.746721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.733902 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-workload-socket\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.746721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.733960 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm62r\" (UniqueName: \"kubernetes.io/projected/f6e02969-c94d-47bf-9b80-278e570b6e0c-kube-api-access-cm62r\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.746721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.735183 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6e02969-c94d-47bf-9b80-278e570b6e0c-istiod-ca-cert\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.794769 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.773125 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-data\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.847722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.847051 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-workload-socket\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.904147 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.904060 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-workload-certs\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.963724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.947079 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-credential-socket\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:21.973733 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.967293 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-token\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:22.035941 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:21.997742 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-podinfo\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:22.130092 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:22.130004 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-envoy\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:22.180087 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:22.180009 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm62r\" (UniqueName: \"kubernetes.io/projected/f6e02969-c94d-47bf-9b80-278e570b6e0c-kube-api-access-cm62r\") pod \"e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:22.292831 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:22.292540 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:22.672777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:22.670771 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx" podUID="21c2897a-3db9-4dfb-9063-8f67092cd30e" containerName="payload-pre-processing" containerID="cri-o://3b151e9ca589089b47e1d755711bca5cf7e427c0ffeb7575298906acaf7efc06" gracePeriod=25 Nov 03 16:54:22.793733 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:22.786492 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m" podUID="694cfc14-a5d4-4111-9266-fe195921dfc7" containerName="payload-processing" containerID="cri-o://e461b8c6c3936a9a8af4e6446cb097f2872ff78bf3c0668ce892f22b2776f69d" gracePeriod=25 Nov 03 16:54:22.793733 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:22.787557 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 16:54:22.802734 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:22.800999 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded" Nov 03 16:54:22.919253 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:22.889403 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:22.965762 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:22.956665 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:22.971528 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:22.971454 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:22.971950 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:22.971824 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" podUID="2f4b9526-8df4-4d0f-a9bc-34824126f4ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Nov 03 16:54:22.979774 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:22.972065 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Nov 03 16:54:23.375256 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:23.375216 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb2f23df-d970-4f16-8292-569a9356374c" path="/var/lib/kubelet/pods/eb2f23df-d970-4f16-8292-569a9356374c/volumes" Nov 03 16:54:23.563765 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:23.517083 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr"] Nov 03 16:54:23.563765 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:23.525084 4947 generic.go:356] "Generic (PLEG): container finished" podID="694cfc14-a5d4-4111-9266-fe195921dfc7" containerID="e461b8c6c3936a9a8af4e6446cb097f2872ff78bf3c0668ce892f22b2776f69d" exitCode=0 Nov 03 16:54:23.563765 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:23.525437 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m" event={"ID":"694cfc14-a5d4-4111-9266-fe195921dfc7","Type":"ContainerDied","Data":"e461b8c6c3936a9a8af4e6446cb097f2872ff78bf3c0668ce892f22b2776f69d"} Nov 03 16:54:23.563765 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:23.531709 4947 generic.go:356] "Generic (PLEG): container finished" podID="21c2897a-3db9-4dfb-9063-8f67092cd30e" containerID="3b151e9ca589089b47e1d755711bca5cf7e427c0ffeb7575298906acaf7efc06" exitCode=0 Nov 03 16:54:23.563765 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:23.531787 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx" event={"ID":"21c2897a-3db9-4dfb-9063-8f67092cd30e","Type":"ContainerDied","Data":"3b151e9ca589089b47e1d755711bca5cf7e427c0ffeb7575298906acaf7efc06"} Nov 03 16:54:23.582743 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:54:23.579649 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e02969_c94d_47bf_9b80_278e570b6e0c.slice/crio-a2e23ed5a31f61bc54a6401338081f075a9b5e4be575eac28a2e360a0c9dd776 WatchSource:0}: Error finding container a2e23ed5a31f61bc54a6401338081f075a9b5e4be575eac28a2e360a0c9dd776: Status 404 returned error can't find the container with id a2e23ed5a31f61bc54a6401338081f075a9b5e4be575eac28a2e360a0c9dd776 Nov 03 16:54:23.616822 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:23.609995 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:54:23.616822 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:23.610090 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:54:23.616822 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:23.610134 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:54:24.236768 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.236730 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m" Nov 03 16:54:24.335786 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.335751 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/694cfc14-a5d4-4111-9266-fe195921dfc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694cfc14-a5d4-4111-9266-fe195921dfc7-plugins-config-volume\") pod \"694cfc14-a5d4-4111-9266-fe195921dfc7\" (UID: \"694cfc14-a5d4-4111-9266-fe195921dfc7\") " Nov 03 16:54:24.336054 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.336030 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/694cfc14-a5d4-4111-9266-fe195921dfc7-kube-api-access-67bmv\" (UniqueName: \"kubernetes.io/projected/694cfc14-a5d4-4111-9266-fe195921dfc7-kube-api-access-67bmv\") pod \"694cfc14-a5d4-4111-9266-fe195921dfc7\" (UID: \"694cfc14-a5d4-4111-9266-fe195921dfc7\") " Nov 03 16:54:24.337100 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.337076 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/694cfc14-a5d4-4111-9266-fe195921dfc7-plugins-config-volume" pod "694cfc14-a5d4-4111-9266-fe195921dfc7" (UID: "694cfc14-a5d4-4111-9266-fe195921dfc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:54:24.340562 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.340536 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/694cfc14-a5d4-4111-9266-fe195921dfc7-kube-api-access-67bmv" pod "694cfc14-a5d4-4111-9266-fe195921dfc7" (UID: "694cfc14-a5d4-4111-9266-fe195921dfc7"). InnerVolumeSpecName "kube-api-access-67bmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:54:24.354137 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.354106 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx" Nov 03 16:54:24.437989 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.437903 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67bmv\" (UniqueName: \"kubernetes.io/projected/694cfc14-a5d4-4111-9266-fe195921dfc7-kube-api-access-67bmv\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:54:24.437989 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.437942 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694cfc14-a5d4-4111-9266-fe195921dfc7-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:54:24.538476 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.538445 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21c2897a-3db9-4dfb-9063-8f67092cd30e-kube-api-access-95rsw\" (UniqueName: \"kubernetes.io/projected/21c2897a-3db9-4dfb-9063-8f67092cd30e-kube-api-access-95rsw\") pod \"21c2897a-3db9-4dfb-9063-8f67092cd30e\" (UID: \"21c2897a-3db9-4dfb-9063-8f67092cd30e\") " Nov 03 16:54:24.538658 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.538582 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21c2897a-3db9-4dfb-9063-8f67092cd30e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c2897a-3db9-4dfb-9063-8f67092cd30e-plugins-config-volume\") pod \"21c2897a-3db9-4dfb-9063-8f67092cd30e\" (UID: \"21c2897a-3db9-4dfb-9063-8f67092cd30e\") " Nov 03 16:54:24.539392 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.539024 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21c2897a-3db9-4dfb-9063-8f67092cd30e-plugins-config-volume" pod "21c2897a-3db9-4dfb-9063-8f67092cd30e" (UID: "21c2897a-3db9-4dfb-9063-8f67092cd30e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:54:24.539533 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.539511 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21c2897a-3db9-4dfb-9063-8f67092cd30e-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:54:24.540672 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.540639 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" event={"ID":"f6e02969-c94d-47bf-9b80-278e570b6e0c","Type":"ContainerStarted","Data":"d2ca55260de1c6e0baa5e5cbd26a937915311a9b0d56598e3a54ab7bb7df0622"} Nov 03 16:54:24.540853 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.540735 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" event={"ID":"f6e02969-c94d-47bf-9b80-278e570b6e0c","Type":"ContainerStarted","Data":"a2e23ed5a31f61bc54a6401338081f075a9b5e4be575eac28a2e360a0c9dd776"} Nov 03 16:54:24.541285 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.541246 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21c2897a-3db9-4dfb-9063-8f67092cd30e-kube-api-access-95rsw" pod "21c2897a-3db9-4dfb-9063-8f67092cd30e" (UID: "21c2897a-3db9-4dfb-9063-8f67092cd30e"). InnerVolumeSpecName "kube-api-access-95rsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:54:24.542406 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.542380 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx" Nov 03 16:54:24.542501 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.542414 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx" event={"ID":"21c2897a-3db9-4dfb-9063-8f67092cd30e","Type":"ContainerDied","Data":"eb3c1ea12ea6661c6caa4be90dbb08dd0de9ff9a7825b242784c221e16c6f4f2"} Nov 03 16:54:24.542544 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.542529 4947 scope.go:122] "RemoveContainer" containerID="3b151e9ca589089b47e1d755711bca5cf7e427c0ffeb7575298906acaf7efc06" Nov 03 16:54:24.544129 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.543946 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m" event={"ID":"694cfc14-a5d4-4111-9266-fe195921dfc7","Type":"ContainerDied","Data":"e2e3de1ea5bc4a4977265ac8529b2282214ce82c83fba47c6c5ce3d9dfbe5fb1"} Nov 03 16:54:24.544129 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.544025 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m" Nov 03 16:54:24.555850 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.555832 4947 scope.go:122] "RemoveContainer" containerID="e461b8c6c3936a9a8af4e6446cb097f2872ff78bf3c0668ce892f22b2776f69d" Nov 03 16:54:24.591529 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.591477 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" podStartSLOduration=3.591463197 podStartE2EDuration="3.591463197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:24.590362856 +0000 UTC m=+2408.179362834" watchObservedRunningTime="2026-11-03 16:54:24.591463197 +0000 UTC m=+2408.180463164" Nov 03 16:54:24.630520 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.630492 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m"] Nov 03 16:54:24.636743 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.636715 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-be8495-78d9bfb545-7c55m"] Nov 03 16:54:24.640521 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.640498 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95rsw\" (UniqueName: \"kubernetes.io/projected/21c2897a-3db9-4dfb-9063-8f67092cd30e-kube-api-access-95rsw\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:54:24.674446 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.674413 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx"] Nov 03 16:54:24.677609 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:24.677586 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-be8495-5bbd7cf95f-njqfx"] Nov 03 16:54:25.293177 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:25.293150 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:25.298442 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:25.298422 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:25.576645 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:25.576609 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:25.577668 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:25.577646 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:54:26.044257 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:26.044236 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" Nov 03 16:54:26.567290 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:26.567255 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21c2897a-3db9-4dfb-9063-8f67092cd30e" path="/var/lib/kubelet/pods/21c2897a-3db9-4dfb-9063-8f67092cd30e/volumes" Nov 03 16:54:26.568049 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:26.568029 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="694cfc14-a5d4-4111-9266-fe195921dfc7" path="/var/lib/kubelet/pods/694cfc14-a5d4-4111-9266-fe195921dfc7/volumes" Nov 03 16:54:29.021359 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.021325 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg"] Nov 03 16:54:29.023777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.021926 4947 cpu_manager.go:414] "containerMap: removing container" podUID="21c2897a-3db9-4dfb-9063-8f67092cd30e" containerName="payload-pre-processing" Nov 03 16:54:29.023777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.021939 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21c2897a-3db9-4dfb-9063-8f67092cd30e" containerName="payload-pre-processing" Nov 03 16:54:29.023777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.021960 4947 cpu_manager.go:414] "containerMap: removing container" podUID="694cfc14-a5d4-4111-9266-fe195921dfc7" containerName="payload-processing" Nov 03 16:54:29.023777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.021964 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="694cfc14-a5d4-4111-9266-fe195921dfc7" containerName="payload-processing" Nov 03 16:54:29.023777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.022056 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="21c2897a-3db9-4dfb-9063-8f67092cd30e" containerName="payload-pre-processing" Nov 03 16:54:29.023777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.022067 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="694cfc14-a5d4-4111-9266-fe195921dfc7" containerName="payload-processing" Nov 03 16:54:29.025365 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.025351 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" Nov 03 16:54:29.043943 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.043920 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-9a80f0-dockercfg-pzp78\"" Nov 03 16:54:29.044666 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.044649 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-9a80f0\"" Nov 03 16:54:29.062589 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.062556 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg"] Nov 03 16:54:29.093342 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.093308 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb"] Nov 03 16:54:29.099891 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.099868 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" Nov 03 16:54:29.355533 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.261991 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb"] Nov 03 16:54:29.355533 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.287417 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h2vp\" (UniqueName: \"kubernetes.io/projected/cf0d4db9-e143-40e6-a677-2c2ea454ac3b-kube-api-access-9h2vp\") pod \"payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg\" (UID: \"cf0d4db9-e143-40e6-a677-2c2ea454ac3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" Nov 03 16:54:29.355533 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.287475 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e214eab6-0f39-415b-8ea9-8f583801475a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb\" (UID: \"e214eab6-0f39-415b-8ea9-8f583801475a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" Nov 03 16:54:29.355533 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.287646 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf0d4db9-e143-40e6-a677-2c2ea454ac3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg\" (UID: \"cf0d4db9-e143-40e6-a677-2c2ea454ac3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" Nov 03 16:54:29.355533 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.287682 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xf5p\" (UniqueName: \"kubernetes.io/projected/e214eab6-0f39-415b-8ea9-8f583801475a-kube-api-access-8xf5p\") pod \"payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb\" (UID: \"e214eab6-0f39-415b-8ea9-8f583801475a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" Nov 03 16:54:29.389174 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.389121 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf0d4db9-e143-40e6-a677-2c2ea454ac3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg\" (UID: \"cf0d4db9-e143-40e6-a677-2c2ea454ac3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" Nov 03 16:54:29.389174 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.389174 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xf5p\" (UniqueName: \"kubernetes.io/projected/e214eab6-0f39-415b-8ea9-8f583801475a-kube-api-access-8xf5p\") pod \"payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb\" (UID: \"e214eab6-0f39-415b-8ea9-8f583801475a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" Nov 03 16:54:29.389473 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.389417 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h2vp\" (UniqueName: \"kubernetes.io/projected/cf0d4db9-e143-40e6-a677-2c2ea454ac3b-kube-api-access-9h2vp\") pod \"payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg\" (UID: \"cf0d4db9-e143-40e6-a677-2c2ea454ac3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" Nov 03 16:54:29.389473 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.389459 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e214eab6-0f39-415b-8ea9-8f583801475a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb\" (UID: \"e214eab6-0f39-415b-8ea9-8f583801475a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" Nov 03 16:54:29.390276 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.390250 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e214eab6-0f39-415b-8ea9-8f583801475a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb\" (UID: \"e214eab6-0f39-415b-8ea9-8f583801475a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" Nov 03 16:54:29.391360 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.391296 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf0d4db9-e143-40e6-a677-2c2ea454ac3b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg\" (UID: \"cf0d4db9-e143-40e6-a677-2c2ea454ac3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" Nov 03 16:54:29.414559 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.414535 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h2vp\" (UniqueName: \"kubernetes.io/projected/cf0d4db9-e143-40e6-a677-2c2ea454ac3b-kube-api-access-9h2vp\") pod \"payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg\" (UID: \"cf0d4db9-e143-40e6-a677-2c2ea454ac3b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" Nov 03 16:54:29.414663 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.414561 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xf5p\" (UniqueName: \"kubernetes.io/projected/e214eab6-0f39-415b-8ea9-8f583801475a-kube-api-access-8xf5p\") pod \"payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb\" (UID: \"e214eab6-0f39-415b-8ea9-8f583801475a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" Nov 03 16:54:29.598233 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.598206 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" Nov 03 16:54:29.639282 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.639253 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" Nov 03 16:54:29.847609 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.847585 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" Nov 03 16:54:29.921887 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.921844 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb"] Nov 03 16:54:29.923139 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:54:29.923099 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode214eab6_0f39_415b_8ea9_8f583801475a.slice/crio-21efee19b07a989f60e135013f1698f0852d8c1eed222cb500328242e784b097 WatchSource:0}: Error finding container 21efee19b07a989f60e135013f1698f0852d8c1eed222cb500328242e784b097: Status 404 returned error can't find the container with id 21efee19b07a989f60e135013f1698f0852d8c1eed222cb500328242e784b097 Nov 03 16:54:29.975609 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:29.971519 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg"] Nov 03 16:54:29.975609 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:54:29.972300 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0d4db9_e143_40e6_a677_2c2ea454ac3b.slice/crio-379b06c0cac3c414f31b62b0c0ac92996f93f1b86667f006a2be5956d71b65df WatchSource:0}: Error finding container 379b06c0cac3c414f31b62b0c0ac92996f93f1b86667f006a2be5956d71b65df: Status 404 returned error can't find the container with id 379b06c0cac3c414f31b62b0c0ac92996f93f1b86667f006a2be5956d71b65df Nov 03 16:54:30.129993 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:54:30.129942 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:54:30.639596 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:30.639560 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" event={"ID":"e214eab6-0f39-415b-8ea9-8f583801475a","Type":"ContainerStarted","Data":"e594230520f3e9fbacb1f2ed94c4e7b4c6ba4d906453aa8aa11a26065a3583d1"} Nov 03 16:54:30.639822 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:30.639625 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" event={"ID":"e214eab6-0f39-415b-8ea9-8f583801475a","Type":"ContainerStarted","Data":"21efee19b07a989f60e135013f1698f0852d8c1eed222cb500328242e784b097"} Nov 03 16:54:30.639822 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:30.639653 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" Nov 03 16:54:30.641179 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:30.641154 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" event={"ID":"cf0d4db9-e143-40e6-a677-2c2ea454ac3b","Type":"ContainerStarted","Data":"f319af464acdfa9275762f313958a8a49048f413a09ba804dddae9c02a3dcc24"} Nov 03 16:54:30.641319 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:30.641199 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" event={"ID":"cf0d4db9-e143-40e6-a677-2c2ea454ac3b","Type":"ContainerStarted","Data":"379b06c0cac3c414f31b62b0c0ac92996f93f1b86667f006a2be5956d71b65df"} Nov 03 16:54:30.641319 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:30.641220 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" Nov 03 16:54:30.686352 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:30.686301 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" podStartSLOduration=1.686287946 podStartE2EDuration="1.686287946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:30.684518769 +0000 UTC m=+2414.273518735" watchObservedRunningTime="2026-11-03 16:54:30.686287946 +0000 UTC m=+2414.275287914" Nov 03 16:54:30.730418 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:30.730369 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" podStartSLOduration=2.7303545 podStartE2EDuration="2.7303545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:30.728241771 +0000 UTC m=+2414.317241750" watchObservedRunningTime="2026-11-03 16:54:30.7303545 +0000 UTC m=+2414.319354469" Nov 03 16:54:31.027614 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:31.027583 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2"] Nov 03 16:54:31.031881 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:31.031859 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" Nov 03 16:54:31.050399 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:31.050374 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-5a5ab1\"" Nov 03 16:54:31.052273 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:31.052252 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-5a5ab1-dockercfg-q9t7m\"" Nov 03 16:54:31.053936 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:31.053912 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2"] Nov 03 16:54:31.293203 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:31.293111 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2\" (UID: \"9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" Nov 03 16:54:31.293778 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:31.293289 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtzzs\" (UniqueName: \"kubernetes.io/projected/9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b-kube-api-access-rtzzs\") pod \"payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2\" (UID: \"9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" Nov 03 16:54:31.394360 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:31.394328 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2\" (UID: \"9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" Nov 03 16:54:31.394536 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:31.394468 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtzzs\" (UniqueName: \"kubernetes.io/projected/9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b-kube-api-access-rtzzs\") pod \"payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2\" (UID: \"9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" Nov 03 16:54:31.395091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:31.395070 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2\" (UID: \"9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" Nov 03 16:54:31.419223 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:31.419191 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtzzs\" (UniqueName: \"kubernetes.io/projected/9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b-kube-api-access-rtzzs\") pod \"payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2\" (UID: \"9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" Nov 03 16:54:31.646309 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:31.646227 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" Nov 03 16:54:31.985327 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:54:31.985297 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f64bb33_aaa3_4312_b1bd_b2fe3d067f2b.slice/crio-da8e5efb402260a3014af6d58e813d374ff7c3e16dfe6030b1a38d1f29b0d0f9 WatchSource:0}: Error finding container da8e5efb402260a3014af6d58e813d374ff7c3e16dfe6030b1a38d1f29b0d0f9: Status 404 returned error can't find the container with id da8e5efb402260a3014af6d58e813d374ff7c3e16dfe6030b1a38d1f29b0d0f9 Nov 03 16:54:31.986117 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:31.986091 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2"] Nov 03 16:54:32.655247 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:32.655217 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" event={"ID":"9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b","Type":"ContainerStarted","Data":"f538fd06f43544a44679e536c9878757c3f599ff9011f023b4f12b42e3df64d7"} Nov 03 16:54:32.655633 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:32.655272 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" event={"ID":"9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b","Type":"ContainerStarted","Data":"da8e5efb402260a3014af6d58e813d374ff7c3e16dfe6030b1a38d1f29b0d0f9"} Nov 03 16:54:32.655633 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:32.655312 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" Nov 03 16:54:32.702684 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:32.702627 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" podStartSLOduration=2.702609285 podStartE2EDuration="2.702609285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:32.700808105 +0000 UTC m=+2416.289808063" watchObservedRunningTime="2026-11-03 16:54:32.702609285 +0000 UTC m=+2416.291609254" Nov 03 16:54:34.574454 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:34.574425 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs"] Nov 03 16:54:39.575521 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:39.574831 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs" podUID="ef532337-ef30-48e6-a371-120051ae7ee0" containerName="payload-pre-processing" containerID="cri-o://392da0a5cd83500c92de07104947547edc342933b7366a45601c71f051fa59de" gracePeriod=25 Nov 03 16:54:39.709311 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:39.709194 4947 generic.go:356] "Generic (PLEG): container finished" podID="ef532337-ef30-48e6-a371-120051ae7ee0" containerID="392da0a5cd83500c92de07104947547edc342933b7366a45601c71f051fa59de" exitCode=0 Nov 03 16:54:39.709311 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:39.709243 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs" event={"ID":"ef532337-ef30-48e6-a371-120051ae7ee0","Type":"ContainerDied","Data":"392da0a5cd83500c92de07104947547edc342933b7366a45601c71f051fa59de"} Nov 03 16:54:39.901861 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:39.901829 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs" Nov 03 16:54:40.053666 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:40.053631 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef532337-ef30-48e6-a371-120051ae7ee0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef532337-ef30-48e6-a371-120051ae7ee0-plugins-config-volume\") pod \"ef532337-ef30-48e6-a371-120051ae7ee0\" (UID: \"ef532337-ef30-48e6-a371-120051ae7ee0\") " Nov 03 16:54:40.053889 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:40.053871 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef532337-ef30-48e6-a371-120051ae7ee0-kube-api-access-hd6wj\" (UniqueName: \"kubernetes.io/projected/ef532337-ef30-48e6-a371-120051ae7ee0-kube-api-access-hd6wj\") pod \"ef532337-ef30-48e6-a371-120051ae7ee0\" (UID: \"ef532337-ef30-48e6-a371-120051ae7ee0\") " Nov 03 16:54:40.054101 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:40.054079 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef532337-ef30-48e6-a371-120051ae7ee0-plugins-config-volume" pod "ef532337-ef30-48e6-a371-120051ae7ee0" (UID: "ef532337-ef30-48e6-a371-120051ae7ee0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:54:40.054493 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:40.054479 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef532337-ef30-48e6-a371-120051ae7ee0-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:54:40.056082 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:40.056062 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef532337-ef30-48e6-a371-120051ae7ee0-kube-api-access-hd6wj" pod "ef532337-ef30-48e6-a371-120051ae7ee0" (UID: "ef532337-ef30-48e6-a371-120051ae7ee0"). InnerVolumeSpecName "kube-api-access-hd6wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:54:40.156139 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:40.156066 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd6wj\" (UniqueName: \"kubernetes.io/projected/ef532337-ef30-48e6-a371-120051ae7ee0-kube-api-access-hd6wj\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:54:40.717311 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:40.717272 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs" event={"ID":"ef532337-ef30-48e6-a371-120051ae7ee0","Type":"ContainerDied","Data":"268c338a56618e168d5561035b58f5a973719cd9fa973c306aa8bc0a6296395f"} Nov 03 16:54:40.717311 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:40.717314 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs" Nov 03 16:54:40.717882 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:40.717384 4947 scope.go:122] "RemoveContainer" containerID="392da0a5cd83500c92de07104947547edc342933b7366a45601c71f051fa59de" Nov 03 16:54:40.775164 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:40.775125 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs"] Nov 03 16:54:40.788003 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:40.787975 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a11d94-789d799fdc-89pzs"] Nov 03 16:54:41.650460 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:41.650427 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" Nov 03 16:54:41.650644 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:41.650495 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" Nov 03 16:54:42.567281 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:42.567249 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef532337-ef30-48e6-a371-120051ae7ee0" path="/var/lib/kubelet/pods/ef532337-ef30-48e6-a371-120051ae7ee0/volumes" Nov 03 16:54:42.819066 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:42.818984 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:54:42.859008 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:42.858967 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x"] Nov 03 16:54:42.860321 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:42.860287 4947 cpu_manager.go:414] "containerMap: removing container" podUID="ef532337-ef30-48e6-a371-120051ae7ee0" containerName="payload-pre-processing" Nov 03 16:54:42.860520 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:42.860503 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef532337-ef30-48e6-a371-120051ae7ee0" containerName="payload-pre-processing" Nov 03 16:54:42.860864 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:42.860845 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef532337-ef30-48e6-a371-120051ae7ee0" containerName="payload-pre-processing" Nov 03 16:54:42.887269 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:42.887241 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x"] Nov 03 16:54:42.887597 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:42.887537 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" Nov 03 16:54:42.912722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:42.908590 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-9a80f0-serving-cert\"" Nov 03 16:54:43.042742 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:43.041067 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab0d416-6e99-420f-9730-2e3b654e354f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x\" (UID: \"cab0d416-6e99-420f-9730-2e3b654e354f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" Nov 03 16:54:43.042742 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:43.041118 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cab0d416-6e99-420f-9730-2e3b654e354f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x\" (UID: \"cab0d416-6e99-420f-9730-2e3b654e354f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" Nov 03 16:54:43.042742 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:43.041187 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xrfg\" (UniqueName: \"kubernetes.io/projected/cab0d416-6e99-420f-9730-2e3b654e354f-kube-api-access-7xrfg\") pod \"maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x\" (UID: \"cab0d416-6e99-420f-9730-2e3b654e354f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" Nov 03 16:54:43.042742 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:43.041349 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cab0d416-6e99-420f-9730-2e3b654e354f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x\" (UID: \"cab0d416-6e99-420f-9730-2e3b654e354f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" Nov 03 16:54:43.143653 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:43.143572 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab0d416-6e99-420f-9730-2e3b654e354f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x\" (UID: \"cab0d416-6e99-420f-9730-2e3b654e354f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" Nov 03 16:54:43.143890 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:43.143872 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cab0d416-6e99-420f-9730-2e3b654e354f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x\" (UID: \"cab0d416-6e99-420f-9730-2e3b654e354f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" Nov 03 16:54:43.144018 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:43.144001 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xrfg\" (UniqueName: \"kubernetes.io/projected/cab0d416-6e99-420f-9730-2e3b654e354f-kube-api-access-7xrfg\") pod \"maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x\" (UID: \"cab0d416-6e99-420f-9730-2e3b654e354f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" Nov 03 16:54:43.144278 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:43.144256 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cab0d416-6e99-420f-9730-2e3b654e354f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x\" (UID: \"cab0d416-6e99-420f-9730-2e3b654e354f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" Nov 03 16:54:43.167198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:43.144447 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab0d416-6e99-420f-9730-2e3b654e354f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x\" (UID: \"cab0d416-6e99-420f-9730-2e3b654e354f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" Nov 03 16:54:43.167198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:43.148664 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cab0d416-6e99-420f-9730-2e3b654e354f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x\" (UID: \"cab0d416-6e99-420f-9730-2e3b654e354f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" Nov 03 16:54:43.167198 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:43.148664 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cab0d416-6e99-420f-9730-2e3b654e354f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x\" (UID: \"cab0d416-6e99-420f-9730-2e3b654e354f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" Nov 03 16:54:43.184175 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:43.184095 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xrfg\" (UniqueName: \"kubernetes.io/projected/cab0d416-6e99-420f-9730-2e3b654e354f-kube-api-access-7xrfg\") pod \"maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x\" (UID: \"cab0d416-6e99-420f-9730-2e3b654e354f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" Nov 03 16:54:43.369592 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:43.369556 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" Nov 03 16:54:43.712780 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:43.712015 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" Nov 03 16:54:43.716358 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:43.716328 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x"] Nov 03 16:54:44.808881 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:44.806561 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" event={"ID":"cab0d416-6e99-420f-9730-2e3b654e354f","Type":"ContainerStarted","Data":"dd91701ac9e15340b42fb8c021bac255e6b1754982131acc8e50ec27651a5cba"} Nov 03 16:54:44.808881 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:44.806639 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" event={"ID":"cab0d416-6e99-420f-9730-2e3b654e354f","Type":"ContainerStarted","Data":"7ac7a0e9e8f20480a580f070ecff5600123f23f981fe9666e4d9bda870cdd8c6"} Nov 03 16:54:44.808881 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:44.808819 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" Nov 03 16:54:44.983724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:44.967520 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" podStartSLOduration=2.665616326 podStartE2EDuration="2.967500616s" totalImagesPullingTime="301.88429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:44.897065354 +0000 UTC m=+2428.486065323" watchObservedRunningTime="2026-11-03 16:54:44.967500616 +0000 UTC m=+2428.556500585" Nov 03 16:54:45.730139 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:45.730093 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:45.917828 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:45.916780 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 16:54:47.343484 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:47.343450 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst"] Nov 03 16:54:47.349951 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:47.349923 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" Nov 03 16:54:47.386578 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:47.386541 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-5a5ab1-serving-cert\"" Nov 03 16:54:47.389012 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:47.388992 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst"] Nov 03 16:54:47.427328 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:47.427285 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/976c6c9d-ae09-43d7-9858-89f5acbf5e20-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst\" (UID: \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" Nov 03 16:54:47.427504 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:47.427422 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27zkv\" (UniqueName: \"kubernetes.io/projected/976c6c9d-ae09-43d7-9858-89f5acbf5e20-kube-api-access-27zkv\") pod \"maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst\" (UID: \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" Nov 03 16:54:47.427504 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:47.427463 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/976c6c9d-ae09-43d7-9858-89f5acbf5e20-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst\" (UID: \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" Nov 03 16:54:47.427763 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:47.427742 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/976c6c9d-ae09-43d7-9858-89f5acbf5e20-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst\" (UID: \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" Nov 03 16:54:47.529611 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:47.529580 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/976c6c9d-ae09-43d7-9858-89f5acbf5e20-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst\" (UID: \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" Nov 03 16:54:47.529896 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:47.529632 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/976c6c9d-ae09-43d7-9858-89f5acbf5e20-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst\" (UID: \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" Nov 03 16:54:47.529896 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:47.529757 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27zkv\" (UniqueName: \"kubernetes.io/projected/976c6c9d-ae09-43d7-9858-89f5acbf5e20-kube-api-access-27zkv\") pod \"maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst\" (UID: \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" Nov 03 16:54:47.529896 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:47.529789 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/976c6c9d-ae09-43d7-9858-89f5acbf5e20-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst\" (UID: \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" Nov 03 16:54:47.530384 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:47.530362 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/976c6c9d-ae09-43d7-9858-89f5acbf5e20-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst\" (UID: \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" Nov 03 16:54:47.532337 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:47.532317 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/976c6c9d-ae09-43d7-9858-89f5acbf5e20-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst\" (UID: \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" Nov 03 16:54:47.532443 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:47.532358 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/976c6c9d-ae09-43d7-9858-89f5acbf5e20-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst\" (UID: \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" Nov 03 16:54:47.554627 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:47.554561 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27zkv\" (UniqueName: \"kubernetes.io/projected/976c6c9d-ae09-43d7-9858-89f5acbf5e20-kube-api-access-27zkv\") pod \"maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst\" (UID: \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" Nov 03 16:54:47.670069 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:47.669982 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" Nov 03 16:54:48.129738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:48.129363 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst"] Nov 03 16:54:48.934817 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:48.932972 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" event={"ID":"976c6c9d-ae09-43d7-9858-89f5acbf5e20","Type":"ContainerStarted","Data":"971a9f9c125cf62a60fe4f7c2eb0b41158d79379941f657c579b2c48f30323d2"} Nov 03 16:54:48.934817 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:48.933037 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" event={"ID":"976c6c9d-ae09-43d7-9858-89f5acbf5e20","Type":"ContainerStarted","Data":"111a16d5143757ed6cc35186f6a06ad2ae0d47c70a700a8f6912cffcd18cb634"} Nov 03 16:54:48.934817 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:48.934107 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" Nov 03 16:54:48.979660 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:48.979599 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" podStartSLOduration=1.5417018759999999 podStartE2EDuration="1.97958516s" totalImagesPullingTime="437.883284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:54:48.978772883 +0000 UTC m=+2432.567772850" watchObservedRunningTime="2026-11-03 16:54:48.97958516 +0000 UTC m=+2432.568585126" Nov 03 16:54:51.901724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:51.901399 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" Nov 03 16:54:55.131510 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:55.131472 4947 generic.go:356] "Generic (PLEG): container finished" podID="2b9c529d-c795-41ac-8145-e126598ecc4e" containerID="038c2cadb91ff28ef9731c3ea519ce7bf9f90a73dc2da6fa18d29f4e484c933b" exitCode=0 Nov 03 16:54:55.132021 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:55.131654 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" event={"ID":"2b9c529d-c795-41ac-8145-e126598ecc4e","Type":"ContainerDied","Data":"038c2cadb91ff28ef9731c3ea519ce7bf9f90a73dc2da6fa18d29f4e484c933b"} Nov 03 16:54:55.132234 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:55.132216 4947 scope.go:122] "RemoveContainer" containerID="038c2cadb91ff28ef9731c3ea519ce7bf9f90a73dc2da6fa18d29f4e484c933b" Nov 03 16:54:55.133911 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:55.133876 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:54:55.134036 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:55.133963 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:54:55.134036 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:55.134012 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:54:55.956331 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:55.956305 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" Nov 03 16:54:56.145158 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:56.145125 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" event={"ID":"2b9c529d-c795-41ac-8145-e126598ecc4e","Type":"ContainerStarted","Data":"45beecef70d2a31d9418a65d7cfdf1824e004a011a5146761e0fa671afc98d01"} Nov 03 16:54:56.384435 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:56.384402 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:54:56.385735 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:56.385692 4947 patch_prober.go:29] interesting pod/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 16:54:56.385875 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:56.385764 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" podUID="2b9c529d-c795-41ac-8145-e126598ecc4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 16:54:57.384247 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:57.384218 4947 patch_prober.go:29] interesting pod/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 16:54:57.384643 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:57.384267 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" podUID="2b9c529d-c795-41ac-8145-e126598ecc4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 16:54:58.383758 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:58.383688 4947 patch_prober.go:29] interesting pod/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 16:54:58.383945 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:58.383787 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" podUID="2b9c529d-c795-41ac-8145-e126598ecc4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 16:54:59.384112 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:59.384072 4947 patch_prober.go:29] interesting pod/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 16:54:59.384540 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:54:59.384149 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" podUID="2b9c529d-c795-41ac-8145-e126598ecc4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 16:55:00.149862 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:55:00.149828 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:55:00.383792 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:00.383756 4947 patch_prober.go:29] interesting pod/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Nov 03 16:55:00.383969 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:00.383810 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" podUID="2b9c529d-c795-41ac-8145-e126598ecc4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Nov 03 16:55:00.999344 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:00.999307 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:55:01.388626 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:01.388548 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:55:01.389866 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:01.389845 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 16:55:02.410791 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:02.410758 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr"] Nov 03 16:55:02.546573 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:02.546535 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" Nov 03 16:55:02.547971 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:02.546408 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr"] Nov 03 16:55:02.584720 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:02.584636 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-c6f303\"" Nov 03 16:55:02.588436 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:02.585068 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-c6f303-dockercfg-gtqb2\"" Nov 03 16:55:02.620290 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:02.620254 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr\" (UID: \"fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" Nov 03 16:55:02.620502 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:02.620486 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqcb4\" (UniqueName: \"kubernetes.io/projected/fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b-kube-api-access-hqcb4\") pod \"payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr\" (UID: \"fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" Nov 03 16:55:02.721459 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:02.721373 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr\" (UID: \"fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" Nov 03 16:55:02.721459 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:02.721429 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqcb4\" (UniqueName: \"kubernetes.io/projected/fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b-kube-api-access-hqcb4\") pod \"payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr\" (UID: \"fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" Nov 03 16:55:02.722164 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:02.722142 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr\" (UID: \"fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" Nov 03 16:55:02.759784 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:02.759754 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqcb4\" (UniqueName: \"kubernetes.io/projected/fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b-kube-api-access-hqcb4\") pod \"payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr\" (UID: \"fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" Nov 03 16:55:02.898606 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:02.898567 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" Nov 03 16:55:03.210371 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:55:03.210339 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab3fb78_c9d8_4ccc_b4eb_ee4425b0121b.slice/crio-c5fd37415aa43ed5b0a022edc921046f51a48735c612ccb10f4ac0b05060919c WatchSource:0}: Error finding container c5fd37415aa43ed5b0a022edc921046f51a48735c612ccb10f4ac0b05060919c: Status 404 returned error can't find the container with id c5fd37415aa43ed5b0a022edc921046f51a48735c612ccb10f4ac0b05060919c Nov 03 16:55:03.211690 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:03.211663 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr"] Nov 03 16:55:03.243850 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:03.243816 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" event={"ID":"fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b","Type":"ContainerStarted","Data":"c5fd37415aa43ed5b0a022edc921046f51a48735c612ccb10f4ac0b05060919c"} Nov 03 16:55:04.251750 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:04.251708 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" event={"ID":"fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b","Type":"ContainerStarted","Data":"573ea6ebd0da011d9bc37a937f20ae11c9c19c1f18cf92109b6beaa213faf6a4"} Nov 03 16:55:04.252245 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:04.251872 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" Nov 03 16:55:04.298580 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:04.298528 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" podStartSLOduration=2.298514726 podStartE2EDuration="2.298514726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:04.295835656 +0000 UTC m=+2447.884835622" watchObservedRunningTime="2026-11-03 16:55:04.298514726 +0000 UTC m=+2447.887514694" Nov 03 16:55:07.891060 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:07.891028 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:55:12.453973 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.450411 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb"] Nov 03 16:55:12.457519 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.457482 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb"] Nov 03 16:55:12.457667 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.457600 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.482927 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.482897 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-aaea00-openshift-default-dockercfg-h7dvk\"" Nov 03 16:55:12.558364 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.558324 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-envoy\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.558520 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.558387 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbj5r\" (UniqueName: \"kubernetes.io/projected/ebe45313-01fc-4f15-b6d0-bf73375602ba-kube-api-access-tbj5r\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.558520 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.558418 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-credential-socket\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.558520 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.558499 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-workload-certs\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.558667 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.558612 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-token\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.558857 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.558831 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe45313-01fc-4f15-b6d0-bf73375602ba-istiod-ca-cert\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.558933 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.558888 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-podinfo\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.558991 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.558940 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-data\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.559049 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.559033 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-workload-socket\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.660622 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.660591 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-token\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.660931 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.660916 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe45313-01fc-4f15-b6d0-bf73375602ba-istiod-ca-cert\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.661057 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.661046 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-podinfo\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.661146 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.661137 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-data\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.661285 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.661274 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-workload-socket\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.661373 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.661365 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-envoy\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.661445 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.661435 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbj5r\" (UniqueName: \"kubernetes.io/projected/ebe45313-01fc-4f15-b6d0-bf73375602ba-kube-api-access-tbj5r\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.661513 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.661502 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-credential-socket\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.661644 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.661634 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-workload-certs\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.662131 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.662107 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-workload-socket\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.662233 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.662170 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-data\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.662686 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.662663 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe45313-01fc-4f15-b6d0-bf73375602ba-istiod-ca-cert\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.662989 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.662968 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-workload-certs\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.663211 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.663191 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-credential-socket\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.667273 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.665435 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-envoy\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.721818 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.711247 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-podinfo\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.741368 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.735113 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbj5r\" (UniqueName: \"kubernetes.io/projected/ebe45313-01fc-4f15-b6d0-bf73375602ba-kube-api-access-tbj5r\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.767359 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.760294 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-token\") pod \"e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:12.784776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:12.779054 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:13.260001 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:13.259976 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb"] Nov 03 16:55:13.260200 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:55:13.260180 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe45313_01fc_4f15_b6d0_bf73375602ba.slice/crio-a1be41f27e8320ba7ea30abcbe23d91ad01b952596d420cd72d0a17fdaad003a WatchSource:0}: Error finding container a1be41f27e8320ba7ea30abcbe23d91ad01b952596d420cd72d0a17fdaad003a: Status 404 returned error can't find the container with id a1be41f27e8320ba7ea30abcbe23d91ad01b952596d420cd72d0a17fdaad003a Nov 03 16:55:13.263195 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:13.263160 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:55:13.263302 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:13.263236 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:55:13.263302 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:13.263275 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:55:13.366753 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:13.366721 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" event={"ID":"ebe45313-01fc-4f15-b6d0-bf73375602ba","Type":"ContainerStarted","Data":"a1be41f27e8320ba7ea30abcbe23d91ad01b952596d420cd72d0a17fdaad003a"} Nov 03 16:55:14.374715 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.374670 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" event={"ID":"ebe45313-01fc-4f15-b6d0-bf73375602ba","Type":"ContainerStarted","Data":"8f3f77e84bc1fba4057b3b98ab0129f46a57c263a466be2c55025b1372ff8d16"} Nov 03 16:55:14.464554 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.464500 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" podStartSLOduration=2.464486532 podStartE2EDuration="2.464486532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:14.461191181 +0000 UTC m=+2458.050191149" watchObservedRunningTime="2026-11-03 16:55:14.464486532 +0000 UTC m=+2458.053486499" Nov 03 16:55:14.632658 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.632571 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg"] Nov 03 16:55:14.636926 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.636856 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.649394 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.649360 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg"] Nov 03 16:55:14.654760 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.654727 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-podinfo\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.655379 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.654791 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-credential-socket\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.655379 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.654838 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-data\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.655379 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.654905 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-token\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.655379 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.655103 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa74cda3-6b8f-4553-ade8-0b4859783708-istiod-ca-cert\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.655379 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.655175 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-envoy\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.655379 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.655215 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-workload-socket\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.655379 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.655276 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghn74\" (UniqueName: \"kubernetes.io/projected/fa74cda3-6b8f-4553-ade8-0b4859783708-kube-api-access-ghn74\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.655379 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.655301 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-workload-certs\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.666237 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.656343 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-1e9d48-openshift-default-dockercfg-ks6zn\"" Nov 03 16:55:14.761795 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.761749 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-podinfo\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.766894 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.766330 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-credential-socket\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.771095 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.767144 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-data\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.771387 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.771367 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-token\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.796792 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.771051 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-data\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.796792 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.766851 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-credential-socket\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.797642 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.797211 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:14.806255 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.806224 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa74cda3-6b8f-4553-ade8-0b4859783708-istiod-ca-cert\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.806546 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.806525 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-envoy\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.806680 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.806666 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-workload-socket\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.806862 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.806841 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghn74\" (UniqueName: \"kubernetes.io/projected/fa74cda3-6b8f-4553-ade8-0b4859783708-kube-api-access-ghn74\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.806965 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.806952 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-workload-certs\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.807475 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.807457 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-workload-certs\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.808256 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.808235 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa74cda3-6b8f-4553-ade8-0b4859783708-istiod-ca-cert\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.809142 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.809121 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-podinfo\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.810891 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.810167 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-workload-socket\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.836992 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.836958 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-envoy\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.867724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.864535 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-token\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.867724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.865218 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghn74\" (UniqueName: \"kubernetes.io/projected/fa74cda3-6b8f-4553-ade8-0b4859783708-kube-api-access-ghn74\") pod \"e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:14.990641 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:14.971299 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:15.021269 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:15.021227 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:15.319020 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:15.318941 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" Nov 03 16:55:15.385097 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:15.385069 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:15.391070 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:15.391046 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:55:15.454381 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:55:15.454356 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa74cda3_6b8f_4553_ade8_0b4859783708.slice/crio-09ade277e0ce14367fe8b9120ac1371e259222c031529c9ed4a20e1693431a8d WatchSource:0}: Error finding container 09ade277e0ce14367fe8b9120ac1371e259222c031529c9ed4a20e1693431a8d: Status 404 returned error can't find the container with id 09ade277e0ce14367fe8b9120ac1371e259222c031529c9ed4a20e1693431a8d Nov 03 16:55:15.456578 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:15.456547 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg"] Nov 03 16:55:15.460412 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:15.459384 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:55:15.460412 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:15.459473 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:55:15.460412 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:15.459520 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:55:16.396747 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:16.396688 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" event={"ID":"fa74cda3-6b8f-4553-ade8-0b4859783708","Type":"ContainerStarted","Data":"11b04b70a269c86cb0a8ab3516d89e0af3d165dac4e8762ffa7de1f18c2324b9"} Nov 03 16:55:16.397189 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:16.396757 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" event={"ID":"fa74cda3-6b8f-4553-ade8-0b4859783708","Type":"ContainerStarted","Data":"09ade277e0ce14367fe8b9120ac1371e259222c031529c9ed4a20e1693431a8d"} Nov 03 16:55:16.448452 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:16.448384 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" podStartSLOduration=2.448368018 podStartE2EDuration="2.448368018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:16.446355124 +0000 UTC m=+2460.035355115" watchObservedRunningTime="2026-11-03 16:55:16.448368018 +0000 UTC m=+2460.037367985" Nov 03 16:55:16.973049 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:16.972990 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:16.980545 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:16.980509 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:17.431737 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:17.421784 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:17.431737 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:17.429103 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:55:19.434184 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:19.434104 4947 generic.go:356] "Generic (PLEG): container finished" podID="f6e02969-c94d-47bf-9b80-278e570b6e0c" containerID="d2ca55260de1c6e0baa5e5cbd26a937915311a9b0d56598e3a54ab7bb7df0622" exitCode=0 Nov 03 16:55:19.434678 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:19.434178 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" event={"ID":"f6e02969-c94d-47bf-9b80-278e570b6e0c","Type":"ContainerDied","Data":"d2ca55260de1c6e0baa5e5cbd26a937915311a9b0d56598e3a54ab7bb7df0622"} Nov 03 16:55:19.434678 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:19.434621 4947 scope.go:122] "RemoveContainer" containerID="d2ca55260de1c6e0baa5e5cbd26a937915311a9b0d56598e3a54ab7bb7df0622" Nov 03 16:55:19.436213 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:19.436183 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:55:19.436312 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:19.436250 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:55:19.436312 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:19.436284 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:55:20.447391 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:20.447359 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" event={"ID":"f6e02969-c94d-47bf-9b80-278e570b6e0c","Type":"ContainerStarted","Data":"029eccfdf65187ac326b67918673a64969dff00bf9698b8c6fcecd4547305156"} Nov 03 16:55:21.289254 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:21.289219 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6"] Nov 03 16:55:21.293039 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:21.293015 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" Nov 03 16:55:21.293643 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:21.293624 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:55:21.294524 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:21.294497 4947 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 16:55:21.294651 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:21.294546 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" podUID="f6e02969-c94d-47bf-9b80-278e570b6e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 16:55:21.322225 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:21.322190 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-1e9d48\"" Nov 03 16:55:21.347348 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:21.346968 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-1e9d48-dockercfg-49ptz\"" Nov 03 16:55:21.354235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:21.354198 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/375a1b54-34a6-4701-b10c-dcf287987ba5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6\" (UID: \"375a1b54-34a6-4701-b10c-dcf287987ba5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" Nov 03 16:55:21.354656 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:21.354630 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6lcc\" (UniqueName: \"kubernetes.io/projected/375a1b54-34a6-4701-b10c-dcf287987ba5-kube-api-access-x6lcc\") pod \"payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6\" (UID: \"375a1b54-34a6-4701-b10c-dcf287987ba5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" Nov 03 16:55:21.368731 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:21.368684 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6"] Nov 03 16:55:21.457045 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:21.456990 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/375a1b54-34a6-4701-b10c-dcf287987ba5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6\" (UID: \"375a1b54-34a6-4701-b10c-dcf287987ba5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" Nov 03 16:55:21.457551 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:21.457197 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6lcc\" (UniqueName: \"kubernetes.io/projected/375a1b54-34a6-4701-b10c-dcf287987ba5-kube-api-access-x6lcc\") pod \"payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6\" (UID: \"375a1b54-34a6-4701-b10c-dcf287987ba5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" Nov 03 16:55:21.459038 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:21.459004 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/375a1b54-34a6-4701-b10c-dcf287987ba5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6\" (UID: \"375a1b54-34a6-4701-b10c-dcf287987ba5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" Nov 03 16:55:21.483031 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:21.482997 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6lcc\" (UniqueName: \"kubernetes.io/projected/375a1b54-34a6-4701-b10c-dcf287987ba5-kube-api-access-x6lcc\") pod \"payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6\" (UID: \"375a1b54-34a6-4701-b10c-dcf287987ba5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" Nov 03 16:55:21.675808 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:21.675252 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" Nov 03 16:55:22.226998 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:22.226964 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6"] Nov 03 16:55:22.227338 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:55:22.227304 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375a1b54_34a6_4701_b10c_dcf287987ba5.slice/crio-f4c03028fc903841ff50fc5c5a259afa626872bef6320b4ba4d958944f817ba8 WatchSource:0}: Error finding container f4c03028fc903841ff50fc5c5a259afa626872bef6320b4ba4d958944f817ba8: Status 404 returned error can't find the container with id f4c03028fc903841ff50fc5c5a259afa626872bef6320b4ba4d958944f817ba8 Nov 03 16:55:22.293990 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:22.293954 4947 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 16:55:22.294141 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:22.294020 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" podUID="f6e02969-c94d-47bf-9b80-278e570b6e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 16:55:22.508749 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:22.504819 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" event={"ID":"375a1b54-34a6-4701-b10c-dcf287987ba5","Type":"ContainerStarted","Data":"c2f47dff1df624818e3dc60f41e5e5ed555bf91037a3c4d1e21986962737975b"} Nov 03 16:55:22.508749 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:22.504901 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" event={"ID":"375a1b54-34a6-4701-b10c-dcf287987ba5","Type":"ContainerStarted","Data":"f4c03028fc903841ff50fc5c5a259afa626872bef6320b4ba4d958944f817ba8"} Nov 03 16:55:22.508749 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:22.505832 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" Nov 03 16:55:22.580659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:22.580601 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" podStartSLOduration=1.58058874 podStartE2EDuration="1.58058874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:22.580023852 +0000 UTC m=+2466.169023826" watchObservedRunningTime="2026-11-03 16:55:22.58058874 +0000 UTC m=+2466.169588708" Nov 03 16:55:23.293850 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:23.293810 4947 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 16:55:23.294062 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:23.293888 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" podUID="f6e02969-c94d-47bf-9b80-278e570b6e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 16:55:24.293433 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:24.293400 4947 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 16:55:24.293881 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:24.293455 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" podUID="f6e02969-c94d-47bf-9b80-278e570b6e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 16:55:25.294307 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:25.294270 4947 patch_prober.go:29] interesting pod/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Nov 03 16:55:25.294742 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:25.294350 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" podUID="f6e02969-c94d-47bf-9b80-278e570b6e0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Nov 03 16:55:25.577369 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:25.577282 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:55:26.297676 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:26.297646 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:55:26.571088 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:26.571017 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 16:55:30.157080 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:55:30.157001 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:55:34.536410 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:34.536378 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" Nov 03 16:55:40.031248 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:40.031210 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv"] Nov 03 16:55:40.037302 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:40.037276 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:55:40.053483 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:40.053457 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-aaea00-serving-cert\"" Nov 03 16:55:40.078142 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:40.078108 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv"] Nov 03 16:55:40.107510 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:40.107479 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9pks\" (UniqueName: \"kubernetes.io/projected/a2e038a9-fecd-45d0-8b43-557fae839699-kube-api-access-j9pks\") pod \"maas-api-e2e-explicit-aaea00-5749b8db47-qggkv\" (UID: \"a2e038a9-fecd-45d0-8b43-557fae839699\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:55:40.107674 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:40.107592 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2e038a9-fecd-45d0-8b43-557fae839699-metrics-certs\") pod \"maas-api-e2e-explicit-aaea00-5749b8db47-qggkv\" (UID: \"a2e038a9-fecd-45d0-8b43-557fae839699\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:55:40.107674 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:40.107666 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2e038a9-fecd-45d0-8b43-557fae839699-maas-api-tls\") pod \"maas-api-e2e-explicit-aaea00-5749b8db47-qggkv\" (UID: \"a2e038a9-fecd-45d0-8b43-557fae839699\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:55:40.107814 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:40.107711 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e038a9-fecd-45d0-8b43-557fae839699-openshift-service-ca\") pod \"maas-api-e2e-explicit-aaea00-5749b8db47-qggkv\" (UID: \"a2e038a9-fecd-45d0-8b43-557fae839699\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:55:40.223532 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:40.217997 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2e038a9-fecd-45d0-8b43-557fae839699-maas-api-tls\") pod \"maas-api-e2e-explicit-aaea00-5749b8db47-qggkv\" (UID: \"a2e038a9-fecd-45d0-8b43-557fae839699\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:55:40.228511 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:40.228474 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e038a9-fecd-45d0-8b43-557fae839699-openshift-service-ca\") pod \"maas-api-e2e-explicit-aaea00-5749b8db47-qggkv\" (UID: \"a2e038a9-fecd-45d0-8b43-557fae839699\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:55:40.229330 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:40.229202 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9pks\" (UniqueName: \"kubernetes.io/projected/a2e038a9-fecd-45d0-8b43-557fae839699-kube-api-access-j9pks\") pod \"maas-api-e2e-explicit-aaea00-5749b8db47-qggkv\" (UID: \"a2e038a9-fecd-45d0-8b43-557fae839699\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:55:40.231476 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:40.231454 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2e038a9-fecd-45d0-8b43-557fae839699-metrics-certs\") pod \"maas-api-e2e-explicit-aaea00-5749b8db47-qggkv\" (UID: \"a2e038a9-fecd-45d0-8b43-557fae839699\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:55:40.232491 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:55:40.223053 4947 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-serving-cert: secret "maas-api-e2e-explicit-aaea00-serving-cert" not found Nov 03 16:55:40.232730 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:55:40.232714 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2e038a9-fecd-45d0-8b43-557fae839699-maas-api-tls podName:a2e038a9-fecd-45d0-8b43-557fae839699 nodeName:}" failed. No retries permitted until 2026-11-03 16:55:40.732670038 +0000 UTC m=+2484.321670027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2e038a9-fecd-45d0-8b43-557fae839699-maas-api-tls") pod "maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" (UID: "a2e038a9-fecd-45d0-8b43-557fae839699") : secret "maas-api-e2e-explicit-aaea00-serving-cert" not found Nov 03 16:55:40.233149 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:40.230918 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e038a9-fecd-45d0-8b43-557fae839699-openshift-service-ca\") pod \"maas-api-e2e-explicit-aaea00-5749b8db47-qggkv\" (UID: \"a2e038a9-fecd-45d0-8b43-557fae839699\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:55:40.261526 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:40.261488 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2e038a9-fecd-45d0-8b43-557fae839699-metrics-certs\") pod \"maas-api-e2e-explicit-aaea00-5749b8db47-qggkv\" (UID: \"a2e038a9-fecd-45d0-8b43-557fae839699\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:55:40.284431 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:40.284145 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9pks\" (UniqueName: \"kubernetes.io/projected/a2e038a9-fecd-45d0-8b43-557fae839699-kube-api-access-j9pks\") pod \"maas-api-e2e-explicit-aaea00-5749b8db47-qggkv\" (UID: \"a2e038a9-fecd-45d0-8b43-557fae839699\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:55:40.741275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:40.741247 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2e038a9-fecd-45d0-8b43-557fae839699-maas-api-tls\") pod \"maas-api-e2e-explicit-aaea00-5749b8db47-qggkv\" (UID: \"a2e038a9-fecd-45d0-8b43-557fae839699\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:55:40.743917 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:40.743890 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2e038a9-fecd-45d0-8b43-557fae839699-maas-api-tls\") pod \"maas-api-e2e-explicit-aaea00-5749b8db47-qggkv\" (UID: \"a2e038a9-fecd-45d0-8b43-557fae839699\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:55:40.964383 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:40.964354 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:55:41.399850 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:41.399027 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv"] Nov 03 16:55:41.668398 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:41.668309 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" event={"ID":"a2e038a9-fecd-45d0-8b43-557fae839699","Type":"ContainerStarted","Data":"89fdcb639f754eba2389164cdafcb8a597e132ed40c49776d817032ecab4a3c9"} Nov 03 16:55:41.889777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:41.883732 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:55:42.676591 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:42.676559 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" event={"ID":"a2e038a9-fecd-45d0-8b43-557fae839699","Type":"ContainerStarted","Data":"41ee801569cab25bf9e3c0a5a4abbfb49ee8f19b7397ea79b473730aa2d7fe66"} Nov 03 16:55:42.677046 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:42.676648 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:55:42.726419 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:42.726355 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" podStartSLOduration=2.430518248 podStartE2EDuration="2.726337898s" totalImagesPullingTime="295.81965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:55:42.723805013 +0000 UTC m=+2486.312804981" watchObservedRunningTime="2026-11-03 16:55:42.726337898 +0000 UTC m=+2486.315337866" Nov 03 16:55:48.713719 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:48.708004 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:55:58.695975 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.695944 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj"] Nov 03 16:55:58.723788 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.720985 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:58.761808 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.754542 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-3b6f35\"/\"default-dockercfg-hdcvc\"" Nov 03 16:55:58.763325 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.763068 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj"] Nov 03 16:55:58.766799 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.766601 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-3b6f35\"/\"auth-test-model-3b6f35-kserve-self-signed-certs\"" Nov 03 16:55:58.766799 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.766647 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-3b6f35\"/\"openshift-service-ca.crt\"" Nov 03 16:55:58.766799 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.766751 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-3b6f35\"/\"kube-root-ca.crt\"" Nov 03 16:55:58.797383 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.797351 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-home\") pod \"auth-test-model-3b6f35-kserve-795fd94d65-cvlbj\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:58.797551 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.797415 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9cbz\" (UniqueName: \"kubernetes.io/projected/abcec173-ab52-4006-b573-f5917ac603c2-kube-api-access-b9cbz\") pod \"auth-test-model-3b6f35-kserve-795fd94d65-cvlbj\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:58.797551 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.797476 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-model-cache\") pod \"auth-test-model-3b6f35-kserve-795fd94d65-cvlbj\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:58.797551 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.797502 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-dshm\") pod \"auth-test-model-3b6f35-kserve-795fd94d65-cvlbj\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:58.797551 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.797520 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abcec173-ab52-4006-b573-f5917ac603c2-tls-certs\") pod \"auth-test-model-3b6f35-kserve-795fd94d65-cvlbj\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:58.797690 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.797604 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-tmp-dir\") pod \"auth-test-model-3b6f35-kserve-795fd94d65-cvlbj\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:58.902723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.902459 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-home\") pod \"auth-test-model-3b6f35-kserve-795fd94d65-cvlbj\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:58.902723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.902541 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9cbz\" (UniqueName: \"kubernetes.io/projected/abcec173-ab52-4006-b573-f5917ac603c2-kube-api-access-b9cbz\") pod \"auth-test-model-3b6f35-kserve-795fd94d65-cvlbj\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:58.902723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.902625 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-model-cache\") pod \"auth-test-model-3b6f35-kserve-795fd94d65-cvlbj\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:58.902723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.902666 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-dshm\") pod \"auth-test-model-3b6f35-kserve-795fd94d65-cvlbj\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:58.902723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.902715 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abcec173-ab52-4006-b573-f5917ac603c2-tls-certs\") pod \"auth-test-model-3b6f35-kserve-795fd94d65-cvlbj\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:58.903160 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.902845 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-tmp-dir\") pod \"auth-test-model-3b6f35-kserve-795fd94d65-cvlbj\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:59.005520 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.903689 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-tmp-dir\") pod \"auth-test-model-3b6f35-kserve-795fd94d65-cvlbj\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:59.005520 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.903966 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-home\") pod \"auth-test-model-3b6f35-kserve-795fd94d65-cvlbj\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:59.005520 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.904482 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-model-cache\") pod \"auth-test-model-3b6f35-kserve-795fd94d65-cvlbj\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:59.005520 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.953412 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abcec173-ab52-4006-b573-f5917ac603c2-tls-certs\") pod \"auth-test-model-3b6f35-kserve-795fd94d65-cvlbj\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:59.005520 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.990897 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-dshm\") pod \"auth-test-model-3b6f35-kserve-795fd94d65-cvlbj\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:59.005520 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:58.991455 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9cbz\" (UniqueName: \"kubernetes.io/projected/abcec173-ab52-4006-b573-f5917ac603c2-kube-api-access-b9cbz\") pod \"auth-test-model-3b6f35-kserve-795fd94d65-cvlbj\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:59.077905 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:59.077822 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:55:59.761330 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:55:59.760241 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj"] Nov 03 16:56:00.062553 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:00.060135 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" event={"ID":"abcec173-ab52-4006-b573-f5917ac603c2","Type":"ContainerStarted","Data":"f8f87679bf301a73988914748e888103075f1e3080753c78f4b17a88ce06d2b1"} Nov 03 16:56:00.062553 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:00.060173 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" event={"ID":"abcec173-ab52-4006-b573-f5917ac603c2","Type":"ContainerStarted","Data":"d039bd707e323d7dbdfb61d9286b3f505ae9839aee4e78670c3c83cd186703ea"} Nov 03 16:56:00.078491 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:00.078463 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:56:00.153723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:00.152690 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" podStartSLOduration=2.152667762 podStartE2EDuration="2.152667762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:56:00.147002284 +0000 UTC m=+2503.736002252" watchObservedRunningTime="2026-11-03 16:56:00.152667762 +0000 UTC m=+2503.741667732" Nov 03 16:56:00.183051 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:56:00.183015 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:56:08.124087 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:08.124053 4947 generic.go:356] "Generic (PLEG): container finished" podID="ebe45313-01fc-4f15-b6d0-bf73375602ba" containerID="8f3f77e84bc1fba4057b3b98ab0129f46a57c263a466be2c55025b1372ff8d16" exitCode=0 Nov 03 16:56:08.124659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:08.124134 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" event={"ID":"ebe45313-01fc-4f15-b6d0-bf73375602ba","Type":"ContainerDied","Data":"8f3f77e84bc1fba4057b3b98ab0129f46a57c263a466be2c55025b1372ff8d16"} Nov 03 16:56:08.124790 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:08.124725 4947 scope.go:122] "RemoveContainer" containerID="8f3f77e84bc1fba4057b3b98ab0129f46a57c263a466be2c55025b1372ff8d16" Nov 03 16:56:08.125993 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:08.125973 4947 generic.go:356] "Generic (PLEG): container finished" podID="fa74cda3-6b8f-4553-ade8-0b4859783708" containerID="11b04b70a269c86cb0a8ab3516d89e0af3d165dac4e8762ffa7de1f18c2324b9" exitCode=0 Nov 03 16:56:08.126091 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:08.126009 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" event={"ID":"fa74cda3-6b8f-4553-ade8-0b4859783708","Type":"ContainerDied","Data":"11b04b70a269c86cb0a8ab3516d89e0af3d165dac4e8762ffa7de1f18c2324b9"} Nov 03 16:56:08.126358 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:08.126346 4947 scope.go:122] "RemoveContainer" containerID="11b04b70a269c86cb0a8ab3516d89e0af3d165dac4e8762ffa7de1f18c2324b9" Nov 03 16:56:08.126559 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:08.126533 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:56:08.126653 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:08.126639 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:56:08.126709 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:08.126684 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:56:08.127903 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:08.127838 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:56:08.128000 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:08.127915 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:56:08.128000 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:08.127969 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 16:56:09.079027 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:09.078996 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:56:09.091500 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:09.091469 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:56:09.165640 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:09.165571 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" event={"ID":"fa74cda3-6b8f-4553-ade8-0b4859783708","Type":"ContainerStarted","Data":"86f2e03f8ef700e19ce127a20158768d266e7d56511763ff3bb5b56498741009"} Nov 03 16:56:09.169248 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:09.169218 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" event={"ID":"ebe45313-01fc-4f15-b6d0-bf73375602ba","Type":"ContainerStarted","Data":"2a6669def861e78881b87086f5be0a32902b6a6282a82547f3aa65bc99c202e1"} Nov 03 16:56:09.181808 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:09.181764 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 16:56:09.780284 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:09.780255 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:56:09.780747 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:09.780530 4947 patch_prober.go:29] interesting pod/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 16:56:09.780747 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:09.780577 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" podUID="ebe45313-01fc-4f15-b6d0-bf73375602ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 16:56:09.972297 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:09.972252 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:56:09.972508 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:09.972374 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 16:56:09.972508 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:09.972423 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" podUID="fa74cda3-6b8f-4553-ade8-0b4859783708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 16:56:10.780892 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:10.780854 4947 patch_prober.go:29] interesting pod/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 16:56:10.781317 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:10.780923 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" podUID="ebe45313-01fc-4f15-b6d0-bf73375602ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 16:56:10.972991 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:10.972956 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 16:56:10.973298 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:10.973258 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" podUID="fa74cda3-6b8f-4553-ade8-0b4859783708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 16:56:11.780516 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:11.780477 4947 patch_prober.go:29] interesting pod/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 16:56:11.780839 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:11.780535 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" podUID="ebe45313-01fc-4f15-b6d0-bf73375602ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 16:56:11.972992 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:11.972954 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 16:56:11.973496 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:11.973020 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" podUID="fa74cda3-6b8f-4553-ade8-0b4859783708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 16:56:12.783890 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:12.783850 4947 patch_prober.go:29] interesting pod/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 16:56:12.784110 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:12.783908 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" podUID="ebe45313-01fc-4f15-b6d0-bf73375602ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 16:56:12.977972 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:12.977767 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 16:56:12.977972 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:12.977842 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" podUID="fa74cda3-6b8f-4553-ade8-0b4859783708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 16:56:13.780220 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:13.780180 4947 patch_prober.go:29] interesting pod/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 16:56:13.780439 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:13.780247 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" podUID="ebe45313-01fc-4f15-b6d0-bf73375602ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 16:56:13.973551 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:13.972675 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 16:56:13.973551 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:13.972763 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" podUID="fa74cda3-6b8f-4553-ade8-0b4859783708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 16:56:14.786193 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:14.786156 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:56:14.786989 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:14.786969 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:56:14.787647 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:14.787626 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:56:15.159308 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:15.159214 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:56:15.160628 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:15.160606 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:56:15.161122 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:15.161106 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 16:56:20.327809 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:20.327768 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:56:22.911467 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:22.911428 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh"] Nov 03 16:56:22.917641 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:22.917618 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:22.938706 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:22.938649 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh"] Nov 03 16:56:22.946574 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:22.946542 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-9a80f0\"/\"default-dockercfg-z59s4\"" Nov 03 16:56:22.946760 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:22.946633 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-9a80f0\"/\"auth-test-model-9a80f0-kserve-self-signed-certs\"" Nov 03 16:56:23.007949 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.007910 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-9a80f0\"/\"kube-root-ca.crt\"" Nov 03 16:56:23.008126 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.008029 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-9a80f0\"/\"openshift-service-ca.crt\"" Nov 03 16:56:23.096048 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.096011 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-tmp-dir\") pod \"auth-test-model-9a80f0-kserve-549b57c99c-c8krh\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.096207 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.096088 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4sqv\" (UniqueName: \"kubernetes.io/projected/fb3ce8a7-ce9a-459d-8580-5fe627e79065-kube-api-access-n4sqv\") pod \"auth-test-model-9a80f0-kserve-549b57c99c-c8krh\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.096207 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.096178 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb3ce8a7-ce9a-459d-8580-5fe627e79065-tls-certs\") pod \"auth-test-model-9a80f0-kserve-549b57c99c-c8krh\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.096348 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.096241 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-dshm\") pod \"auth-test-model-9a80f0-kserve-549b57c99c-c8krh\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.096431 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.096339 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-home\") pod \"auth-test-model-9a80f0-kserve-549b57c99c-c8krh\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.096524 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.096504 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-model-cache\") pod \"auth-test-model-9a80f0-kserve-549b57c99c-c8krh\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.199777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.199576 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-tmp-dir\") pod \"auth-test-model-9a80f0-kserve-549b57c99c-c8krh\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.199777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.199751 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4sqv\" (UniqueName: \"kubernetes.io/projected/fb3ce8a7-ce9a-459d-8580-5fe627e79065-kube-api-access-n4sqv\") pod \"auth-test-model-9a80f0-kserve-549b57c99c-c8krh\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.200042 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.199796 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb3ce8a7-ce9a-459d-8580-5fe627e79065-tls-certs\") pod \"auth-test-model-9a80f0-kserve-549b57c99c-c8krh\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.200042 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.199858 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-dshm\") pod \"auth-test-model-9a80f0-kserve-549b57c99c-c8krh\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.200042 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.199890 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-home\") pod \"auth-test-model-9a80f0-kserve-549b57c99c-c8krh\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.200042 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.199999 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-model-cache\") pod \"auth-test-model-9a80f0-kserve-549b57c99c-c8krh\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.203724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.200456 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-model-cache\") pod \"auth-test-model-9a80f0-kserve-549b57c99c-c8krh\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.203724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.200709 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-tmp-dir\") pod \"auth-test-model-9a80f0-kserve-549b57c99c-c8krh\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.210719 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.206644 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-home\") pod \"auth-test-model-9a80f0-kserve-549b57c99c-c8krh\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.226722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.220403 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb3ce8a7-ce9a-459d-8580-5fe627e79065-tls-certs\") pod \"auth-test-model-9a80f0-kserve-549b57c99c-c8krh\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.226722 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.221875 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-dshm\") pod \"auth-test-model-9a80f0-kserve-549b57c99c-c8krh\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.240877 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.232147 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4sqv\" (UniqueName: \"kubernetes.io/projected/fb3ce8a7-ce9a-459d-8580-5fe627e79065-kube-api-access-n4sqv\") pod \"auth-test-model-9a80f0-kserve-549b57c99c-c8krh\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.240877 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.237028 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:23.616665 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:56:23.616630 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb3ce8a7_ce9a_459d_8580_5fe627e79065.slice/crio-67bc0d2981936fc5a509ab6ea11d015887583f4db2c7e18c28f876530fb9ffc5 WatchSource:0}: Error finding container 67bc0d2981936fc5a509ab6ea11d015887583f4db2c7e18c28f876530fb9ffc5: Status 404 returned error can't find the container with id 67bc0d2981936fc5a509ab6ea11d015887583f4db2c7e18c28f876530fb9ffc5 Nov 03 16:56:23.619933 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:23.619910 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh"] Nov 03 16:56:24.357644 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:24.357613 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" event={"ID":"fb3ce8a7-ce9a-459d-8580-5fe627e79065","Type":"ContainerStarted","Data":"aa17bd96bbe11b1f8ee5f232671796e1cb88c1992e82ae2a9754c31970b6fd95"} Nov 03 16:56:24.358059 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:24.357656 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" event={"ID":"fb3ce8a7-ce9a-459d-8580-5fe627e79065","Type":"ContainerStarted","Data":"67bc0d2981936fc5a509ab6ea11d015887583f4db2c7e18c28f876530fb9ffc5"} Nov 03 16:56:24.405503 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:24.405449 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" podStartSLOduration=2.405434212 podStartE2EDuration="2.405434212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:56:24.401920723 +0000 UTC m=+2527.990920691" watchObservedRunningTime="2026-11-03 16:56:24.405434212 +0000 UTC m=+2527.994434179" Nov 03 16:56:25.237938 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:25.237883 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:30.131890 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:56:30.131852 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:56:33.237875 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:33.237839 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:33.250299 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:33.250272 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:56:33.437488 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:56:33.437462 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 16:57:00.131107 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:57:00.131020 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:57:00.239793 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.239756 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk"] Nov 03 16:57:00.250049 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.250013 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.267031 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.266987 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk"] Nov 03 16:57:00.275677 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.275637 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-001802\"/\"kube-root-ca.crt\"" Nov 03 16:57:00.279579 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.279553 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-001802\"/\"openshift-service-ca.crt\"" Nov 03 16:57:00.279579 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.279564 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-001802\"/\"auth-test-model-001802-kserve-self-signed-certs\"" Nov 03 16:57:00.281133 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.281097 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-001802\"/\"default-dockercfg-lljrr\"" Nov 03 16:57:00.328630 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.328595 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg8rp\" (UniqueName: \"kubernetes.io/projected/ef66bd84-620e-40a2-9c81-489eb827e636-kube-api-access-hg8rp\") pod \"auth-test-model-001802-kserve-68dcf67cbf-2wwtk\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.328833 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.328660 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-dshm\") pod \"auth-test-model-001802-kserve-68dcf67cbf-2wwtk\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.328833 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.328785 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-home\") pod \"auth-test-model-001802-kserve-68dcf67cbf-2wwtk\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.328919 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.328834 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef66bd84-620e-40a2-9c81-489eb827e636-tls-certs\") pod \"auth-test-model-001802-kserve-68dcf67cbf-2wwtk\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.328955 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.328917 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-model-cache\") pod \"auth-test-model-001802-kserve-68dcf67cbf-2wwtk\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.329045 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.329029 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-tmp-dir\") pod \"auth-test-model-001802-kserve-68dcf67cbf-2wwtk\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.430368 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.430271 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-dshm\") pod \"auth-test-model-001802-kserve-68dcf67cbf-2wwtk\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.430368 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.430345 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-home\") pod \"auth-test-model-001802-kserve-68dcf67cbf-2wwtk\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.430368 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.430368 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef66bd84-620e-40a2-9c81-489eb827e636-tls-certs\") pod \"auth-test-model-001802-kserve-68dcf67cbf-2wwtk\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.430651 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.430435 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-model-cache\") pod \"auth-test-model-001802-kserve-68dcf67cbf-2wwtk\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.430651 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.430545 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-tmp-dir\") pod \"auth-test-model-001802-kserve-68dcf67cbf-2wwtk\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.430797 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.430717 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg8rp\" (UniqueName: \"kubernetes.io/projected/ef66bd84-620e-40a2-9c81-489eb827e636-kube-api-access-hg8rp\") pod \"auth-test-model-001802-kserve-68dcf67cbf-2wwtk\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.430893 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.430868 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-home\") pod \"auth-test-model-001802-kserve-68dcf67cbf-2wwtk\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.430965 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.430917 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-model-cache\") pod \"auth-test-model-001802-kserve-68dcf67cbf-2wwtk\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.431217 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.431194 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-tmp-dir\") pod \"auth-test-model-001802-kserve-68dcf67cbf-2wwtk\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.432789 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.432767 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-dshm\") pod \"auth-test-model-001802-kserve-68dcf67cbf-2wwtk\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.433027 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.433009 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef66bd84-620e-40a2-9c81-489eb827e636-tls-certs\") pod \"auth-test-model-001802-kserve-68dcf67cbf-2wwtk\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.459897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.459867 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg8rp\" (UniqueName: \"kubernetes.io/projected/ef66bd84-620e-40a2-9c81-489eb827e636-kube-api-access-hg8rp\") pod \"auth-test-model-001802-kserve-68dcf67cbf-2wwtk\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.599723 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.596198 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:00.989527 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:00.989447 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:57:01.031874 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:01.031847 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk"] Nov 03 16:57:01.033507 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:57:01.033480 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef66bd84_620e_40a2_9c81_489eb827e636.slice/crio-9bf8547f3c6eb66c6d4dcf5cbd3e31ff082832ad51b674ea01ce019c641a2f5e WatchSource:0}: Error finding container 9bf8547f3c6eb66c6d4dcf5cbd3e31ff082832ad51b674ea01ce019c641a2f5e: Status 404 returned error can't find the container with id 9bf8547f3c6eb66c6d4dcf5cbd3e31ff082832ad51b674ea01ce019c641a2f5e Nov 03 16:57:01.747632 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:01.747595 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" event={"ID":"ef66bd84-620e-40a2-9c81-489eb827e636","Type":"ContainerStarted","Data":"a54389217dca4f94beb5a86f71f970751c9e422175a9ec2e8c17f901f1f0b898"} Nov 03 16:57:01.747632 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:01.747638 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" event={"ID":"ef66bd84-620e-40a2-9c81-489eb827e636","Type":"ContainerStarted","Data":"9bf8547f3c6eb66c6d4dcf5cbd3e31ff082832ad51b674ea01ce019c641a2f5e"} Nov 03 16:57:01.797038 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:01.796989 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" podStartSLOduration=1.796975231 podStartE2EDuration="1.796975231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:57:01.793273766 +0000 UTC m=+2565.382273734" watchObservedRunningTime="2026-11-03 16:57:01.796975231 +0000 UTC m=+2565.385975198" Nov 03 16:57:02.597089 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:02.597051 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:10.597769 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:10.597727 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:10.610071 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:10.610045 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:10.825777 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:10.825748 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 16:57:30.131707 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:57:30.131669 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:57:44.990418 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:44.990382 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:57:48.355243 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:48.355207 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv"] Nov 03 16:57:48.355781 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:48.355480 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" podUID="a2e038a9-fecd-45d0-8b43-557fae839699" containerName="maas-api" containerID="cri-o://41ee801569cab25bf9e3c0a5a4abbfb49ee8f19b7397ea79b473730aa2d7fe66" gracePeriod=30 Nov 03 16:57:48.899910 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:48.899876 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:57:48.941897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:48.936351 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2e038a9-fecd-45d0-8b43-557fae839699-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e038a9-fecd-45d0-8b43-557fae839699-openshift-service-ca\") pod \"a2e038a9-fecd-45d0-8b43-557fae839699\" (UID: \"a2e038a9-fecd-45d0-8b43-557fae839699\") " Nov 03 16:57:48.941897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:48.936413 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e038a9-fecd-45d0-8b43-557fae839699-kube-api-access-j9pks\" (UniqueName: \"kubernetes.io/projected/a2e038a9-fecd-45d0-8b43-557fae839699-kube-api-access-j9pks\") pod \"a2e038a9-fecd-45d0-8b43-557fae839699\" (UID: \"a2e038a9-fecd-45d0-8b43-557fae839699\") " Nov 03 16:57:48.941897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:48.936584 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2e038a9-fecd-45d0-8b43-557fae839699-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2e038a9-fecd-45d0-8b43-557fae839699-maas-api-tls\") pod \"a2e038a9-fecd-45d0-8b43-557fae839699\" (UID: \"a2e038a9-fecd-45d0-8b43-557fae839699\") " Nov 03 16:57:48.941897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:48.936614 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2e038a9-fecd-45d0-8b43-557fae839699-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2e038a9-fecd-45d0-8b43-557fae839699-metrics-certs\") pod \"a2e038a9-fecd-45d0-8b43-557fae839699\" (UID: \"a2e038a9-fecd-45d0-8b43-557fae839699\") " Nov 03 16:57:48.941897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:48.938867 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e038a9-fecd-45d0-8b43-557fae839699-openshift-service-ca" pod "a2e038a9-fecd-45d0-8b43-557fae839699" (UID: "a2e038a9-fecd-45d0-8b43-557fae839699"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:57:48.964774 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:48.964647 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2e038a9-fecd-45d0-8b43-557fae839699-maas-api-tls" pod "a2e038a9-fecd-45d0-8b43-557fae839699" (UID: "a2e038a9-fecd-45d0-8b43-557fae839699"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:57:48.980770 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:48.972635 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2e038a9-fecd-45d0-8b43-557fae839699-metrics-certs" pod "a2e038a9-fecd-45d0-8b43-557fae839699" (UID: "a2e038a9-fecd-45d0-8b43-557fae839699"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:57:48.993932 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:48.992886 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e038a9-fecd-45d0-8b43-557fae839699-kube-api-access-j9pks" pod "a2e038a9-fecd-45d0-8b43-557fae839699" (UID: "a2e038a9-fecd-45d0-8b43-557fae839699"). InnerVolumeSpecName "kube-api-access-j9pks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:57:49.039902 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:49.039867 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2e038a9-fecd-45d0-8b43-557fae839699-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:57:49.039902 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:49.039900 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9pks\" (UniqueName: \"kubernetes.io/projected/a2e038a9-fecd-45d0-8b43-557fae839699-kube-api-access-j9pks\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:57:49.040127 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:49.039913 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2e038a9-fecd-45d0-8b43-557fae839699-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:57:49.040127 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:49.039931 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2e038a9-fecd-45d0-8b43-557fae839699-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:57:49.169136 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:49.169046 4947 generic.go:356] "Generic (PLEG): container finished" podID="a2e038a9-fecd-45d0-8b43-557fae839699" containerID="41ee801569cab25bf9e3c0a5a4abbfb49ee8f19b7397ea79b473730aa2d7fe66" exitCode=0 Nov 03 16:57:49.173466 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:49.169453 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" event={"ID":"a2e038a9-fecd-45d0-8b43-557fae839699","Type":"ContainerDied","Data":"41ee801569cab25bf9e3c0a5a4abbfb49ee8f19b7397ea79b473730aa2d7fe66"} Nov 03 16:57:49.173802 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:49.173744 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" event={"ID":"a2e038a9-fecd-45d0-8b43-557fae839699","Type":"ContainerDied","Data":"89fdcb639f754eba2389164cdafcb8a597e132ed40c49776d817032ecab4a3c9"} Nov 03 16:57:49.173991 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:49.173975 4947 scope.go:122] "RemoveContainer" containerID="41ee801569cab25bf9e3c0a5a4abbfb49ee8f19b7397ea79b473730aa2d7fe66" Nov 03 16:57:49.174353 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:49.174324 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" Nov 03 16:57:49.462206 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:49.462180 4947 scope.go:122] "RemoveContainer" containerID="41ee801569cab25bf9e3c0a5a4abbfb49ee8f19b7397ea79b473730aa2d7fe66" Nov 03 16:57:49.462587 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:57:49.462534 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41ee801569cab25bf9e3c0a5a4abbfb49ee8f19b7397ea79b473730aa2d7fe66\": container with ID starting with 41ee801569cab25bf9e3c0a5a4abbfb49ee8f19b7397ea79b473730aa2d7fe66 not found: ID does not exist" containerID="41ee801569cab25bf9e3c0a5a4abbfb49ee8f19b7397ea79b473730aa2d7fe66" Nov 03 16:57:49.462587 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:49.462562 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41ee801569cab25bf9e3c0a5a4abbfb49ee8f19b7397ea79b473730aa2d7fe66"} err="failed to get container status \"41ee801569cab25bf9e3c0a5a4abbfb49ee8f19b7397ea79b473730aa2d7fe66\": rpc error: code = NotFound desc = could not find container \"41ee801569cab25bf9e3c0a5a4abbfb49ee8f19b7397ea79b473730aa2d7fe66\": container with ID starting with 41ee801569cab25bf9e3c0a5a4abbfb49ee8f19b7397ea79b473730aa2d7fe66 not found: ID does not exist" Nov 03 16:57:49.491899 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:49.491013 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv"] Nov 03 16:57:49.513615 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:49.513571 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv"] Nov 03 16:57:50.568461 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:50.568428 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2e038a9-fecd-45d0-8b43-557fae839699" path="/var/lib/kubelet/pods/a2e038a9-fecd-45d0-8b43-557fae839699/volumes" Nov 03 16:57:51.718401 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:51.718362 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aaea00-5749b8db47-qggkv" podUID="a2e038a9-fecd-45d0-8b43-557fae839699" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.161:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 16:57:55.330743 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:55.330667 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq"] Nov 03 16:57:55.331569 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:55.331550 4947 cpu_manager.go:414] "containerMap: removing container" podUID="a2e038a9-fecd-45d0-8b43-557fae839699" containerName="maas-api" Nov 03 16:57:55.331641 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:55.331571 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e038a9-fecd-45d0-8b43-557fae839699" containerName="maas-api" Nov 03 16:57:55.331745 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:55.331733 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e038a9-fecd-45d0-8b43-557fae839699" containerName="maas-api" Nov 03 16:57:55.336778 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:55.336757 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq" Nov 03 16:57:55.357112 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:55.357081 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq"] Nov 03 16:57:55.365913 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:55.365894 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-c1837e-dockercfg-jxj89\"" Nov 03 16:57:55.398544 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:55.398109 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-c1837e\"" Nov 03 16:57:55.439799 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:55.430127 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae217bd-e9a6-497c-bc9a-54cfc421db75-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq\" (UID: \"1ae217bd-e9a6-497c-bc9a-54cfc421db75\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq" Nov 03 16:57:55.439799 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:55.430301 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7thm\" (UniqueName: \"kubernetes.io/projected/1ae217bd-e9a6-497c-bc9a-54cfc421db75-kube-api-access-n7thm\") pod \"payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq\" (UID: \"1ae217bd-e9a6-497c-bc9a-54cfc421db75\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq" Nov 03 16:57:55.576721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:55.559149 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae217bd-e9a6-497c-bc9a-54cfc421db75-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq\" (UID: \"1ae217bd-e9a6-497c-bc9a-54cfc421db75\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq" Nov 03 16:57:55.576721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:55.559295 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7thm\" (UniqueName: \"kubernetes.io/projected/1ae217bd-e9a6-497c-bc9a-54cfc421db75-kube-api-access-n7thm\") pod \"payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq\" (UID: \"1ae217bd-e9a6-497c-bc9a-54cfc421db75\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq" Nov 03 16:57:55.576721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:55.560765 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae217bd-e9a6-497c-bc9a-54cfc421db75-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq\" (UID: \"1ae217bd-e9a6-497c-bc9a-54cfc421db75\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq" Nov 03 16:57:55.625099 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:55.624413 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7thm\" (UniqueName: \"kubernetes.io/projected/1ae217bd-e9a6-497c-bc9a-54cfc421db75-kube-api-access-n7thm\") pod \"payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq\" (UID: \"1ae217bd-e9a6-497c-bc9a-54cfc421db75\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq" Nov 03 16:57:55.650462 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:55.650425 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq" Nov 03 16:57:55.976144 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:55.976118 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq"] Nov 03 16:57:56.277905 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:56.277874 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq" event={"ID":"1ae217bd-e9a6-497c-bc9a-54cfc421db75","Type":"ContainerStarted","Data":"ffef7990826c855bcb3b15429a4645dec9430c4494bd3d86c8255d10624d247e"} Nov 03 16:57:56.278090 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:56.277922 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq" event={"ID":"1ae217bd-e9a6-497c-bc9a-54cfc421db75","Type":"ContainerStarted","Data":"7e6bcfb651e8d2e4b8084fe25c9a177f7071452910eb04e676c4a4e7cd186a10"} Nov 03 16:57:56.278090 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:56.277952 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq" Nov 03 16:57:56.328123 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:56.328071 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq" podStartSLOduration=1.328056558 podStartE2EDuration="1.328056558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:57:56.324373003 +0000 UTC m=+2619.913372972" watchObservedRunningTime="2026-11-03 16:57:56.328056558 +0000 UTC m=+2619.917056525" Nov 03 16:57:59.157789 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:59.154819 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb"] Nov 03 16:57:59.157789 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:59.155069 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" podUID="ebe45313-01fc-4f15-b6d0-bf73375602ba" containerName="istio-proxy" containerID="cri-o://2a6669def861e78881b87086f5be0a32902b6a6282a82547f3aa65bc99c202e1" gracePeriod=30 Nov 03 16:57:59.789321 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:59.789281 4947 patch_prober.go:29] interesting pod/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 16:57:59.789513 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:57:59.789354 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" podUID="ebe45313-01fc-4f15-b6d0-bf73375602ba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 16:58:00.130119 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:58:00.130040 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:58:04.385482 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.385444 4947 generic.go:356] "Generic (PLEG): container finished" podID="ebe45313-01fc-4f15-b6d0-bf73375602ba" containerID="2a6669def861e78881b87086f5be0a32902b6a6282a82547f3aa65bc99c202e1" exitCode=0 Nov 03 16:58:04.386023 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.385518 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" event={"ID":"ebe45313-01fc-4f15-b6d0-bf73375602ba","Type":"ContainerDied","Data":"2a6669def861e78881b87086f5be0a32902b6a6282a82547f3aa65bc99c202e1"} Nov 03 16:58:04.386023 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.385598 4947 scope.go:122] "RemoveContainer" containerID="8f3f77e84bc1fba4057b3b98ab0129f46a57c263a466be2c55025b1372ff8d16" Nov 03 16:58:04.438982 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.438927 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:58:04.482869 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.482837 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-podinfo\") pod \"ebe45313-01fc-4f15-b6d0-bf73375602ba\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " Nov 03 16:58:04.483004 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.482952 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-data\") pod \"ebe45313-01fc-4f15-b6d0-bf73375602ba\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " Nov 03 16:58:04.483069 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.483032 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebe45313-01fc-4f15-b6d0-bf73375602ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe45313-01fc-4f15-b6d0-bf73375602ba-istiod-ca-cert\") pod \"ebe45313-01fc-4f15-b6d0-bf73375602ba\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " Nov 03 16:58:04.483121 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.483089 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe45313-01fc-4f15-b6d0-bf73375602ba-kube-api-access-tbj5r\" (UniqueName: \"kubernetes.io/projected/ebe45313-01fc-4f15-b6d0-bf73375602ba-kube-api-access-tbj5r\") pod \"ebe45313-01fc-4f15-b6d0-bf73375602ba\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " Nov 03 16:58:04.483179 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.483129 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-workload-certs\") pod \"ebe45313-01fc-4f15-b6d0-bf73375602ba\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " Nov 03 16:58:04.483179 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.483173 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-workload-socket\") pod \"ebe45313-01fc-4f15-b6d0-bf73375602ba\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " Nov 03 16:58:04.483288 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.483246 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-credential-socket\") pod \"ebe45313-01fc-4f15-b6d0-bf73375602ba\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " Nov 03 16:58:04.483412 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.483240 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-data" pod "ebe45313-01fc-4f15-b6d0-bf73375602ba" (UID: "ebe45313-01fc-4f15-b6d0-bf73375602ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:04.483487 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.483411 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-workload-socket" pod "ebe45313-01fc-4f15-b6d0-bf73375602ba" (UID: "ebe45313-01fc-4f15-b6d0-bf73375602ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:04.483487 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.483428 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-workload-certs" pod "ebe45313-01fc-4f15-b6d0-bf73375602ba" (UID: "ebe45313-01fc-4f15-b6d0-bf73375602ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:04.483487 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.483442 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-envoy\") pod \"ebe45313-01fc-4f15-b6d0-bf73375602ba\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " Nov 03 16:58:04.483487 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.483468 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-credential-socket" pod "ebe45313-01fc-4f15-b6d0-bf73375602ba" (UID: "ebe45313-01fc-4f15-b6d0-bf73375602ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:04.483651 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.483521 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe45313-01fc-4f15-b6d0-bf73375602ba-istiod-ca-cert" pod "ebe45313-01fc-4f15-b6d0-bf73375602ba" (UID: "ebe45313-01fc-4f15-b6d0-bf73375602ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:58:04.483964 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.483940 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-token\" (UniqueName: \"kubernetes.io/projected/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-token\") pod \"ebe45313-01fc-4f15-b6d0-bf73375602ba\" (UID: \"ebe45313-01fc-4f15-b6d0-bf73375602ba\") " Nov 03 16:58:04.484515 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.484495 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:58:04.484603 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.484517 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:58:04.484603 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.484530 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:58:04.484603 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.484542 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:58:04.484603 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.484553 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebe45313-01fc-4f15-b6d0-bf73375602ba-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:58:04.485475 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.485438 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe45313-01fc-4f15-b6d0-bf73375602ba-kube-api-access-tbj5r" pod "ebe45313-01fc-4f15-b6d0-bf73375602ba" (UID: "ebe45313-01fc-4f15-b6d0-bf73375602ba"). InnerVolumeSpecName "kube-api-access-tbj5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:58:04.485650 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.485632 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-envoy" pod "ebe45313-01fc-4f15-b6d0-bf73375602ba" (UID: "ebe45313-01fc-4f15-b6d0-bf73375602ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:04.485796 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.485776 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-podinfo" pod "ebe45313-01fc-4f15-b6d0-bf73375602ba" (UID: "ebe45313-01fc-4f15-b6d0-bf73375602ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:58:04.486146 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.486123 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-token" pod "ebe45313-01fc-4f15-b6d0-bf73375602ba" (UID: "ebe45313-01fc-4f15-b6d0-bf73375602ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:58:04.586355 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.586328 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbj5r\" (UniqueName: \"kubernetes.io/projected/ebe45313-01fc-4f15-b6d0-bf73375602ba-kube-api-access-tbj5r\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:58:04.586355 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.586357 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:58:04.586509 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.586366 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:58:04.586509 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:04.586373 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebe45313-01fc-4f15-b6d0-bf73375602ba-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:58:05.393818 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:05.393791 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" Nov 03 16:58:05.394406 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:05.393830 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb" event={"ID":"ebe45313-01fc-4f15-b6d0-bf73375602ba","Type":"ContainerDied","Data":"a1be41f27e8320ba7ea30abcbe23d91ad01b952596d420cd72d0a17fdaad003a"} Nov 03 16:58:05.394406 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:05.393893 4947 scope.go:122] "RemoveContainer" containerID="2a6669def861e78881b87086f5be0a32902b6a6282a82547f3aa65bc99c202e1" Nov 03 16:58:05.464757 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:05.464710 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb"] Nov 03 16:58:05.477983 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:05.477956 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-aaea00-openshift-default-7488d7db6f-q2kvb"] Nov 03 16:58:06.567478 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:06.567441 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebe45313-01fc-4f15-b6d0-bf73375602ba" path="/var/lib/kubelet/pods/ebe45313-01fc-4f15-b6d0-bf73375602ba/volumes" Nov 03 16:58:07.288073 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:07.288046 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq" Nov 03 16:58:12.601213 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.601178 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn"] Nov 03 16:58:12.601885 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.601872 4947 cpu_manager.go:414] "containerMap: removing container" podUID="ebe45313-01fc-4f15-b6d0-bf73375602ba" containerName="istio-proxy" Nov 03 16:58:12.601945 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.601887 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe45313-01fc-4f15-b6d0-bf73375602ba" containerName="istio-proxy" Nov 03 16:58:12.601945 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.601903 4947 cpu_manager.go:414] "containerMap: removing container" podUID="ebe45313-01fc-4f15-b6d0-bf73375602ba" containerName="istio-proxy" Nov 03 16:58:12.601945 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.601908 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe45313-01fc-4f15-b6d0-bf73375602ba" containerName="istio-proxy" Nov 03 16:58:12.602042 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.602030 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe45313-01fc-4f15-b6d0-bf73375602ba" containerName="istio-proxy" Nov 03 16:58:12.602042 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.602040 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe45313-01fc-4f15-b6d0-bf73375602ba" containerName="istio-proxy" Nov 03 16:58:12.608168 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.607330 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.628906 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.628859 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn"] Nov 03 16:58:12.637468 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.637435 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-43b1d1\"/\"kube-root-ca.crt\"" Nov 03 16:58:12.637649 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.637455 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-43b1d1\"/\"openshift-service-ca.crt\"" Nov 03 16:58:12.640334 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.640311 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-43b1d1\"/\"noten-model-43b1d1-kserve-self-signed-certs\"" Nov 03 16:58:12.640334 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.640312 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-43b1d1\"/\"default-dockercfg-w96pp\"" Nov 03 16:58:12.737651 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.737608 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c49e65-ef1f-4f1a-afa9-54d21de21725-tls-certs\") pod \"noten-model-43b1d1-kserve-78fcdfd97c-685tn\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.737846 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.737760 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-tmp-dir\") pod \"noten-model-43b1d1-kserve-78fcdfd97c-685tn\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.737846 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.737790 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjbwk\" (UniqueName: \"kubernetes.io/projected/e4c49e65-ef1f-4f1a-afa9-54d21de21725-kube-api-access-zjbwk\") pod \"noten-model-43b1d1-kserve-78fcdfd97c-685tn\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.737846 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.737815 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-dshm\") pod \"noten-model-43b1d1-kserve-78fcdfd97c-685tn\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.738052 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.738030 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-home\") pod \"noten-model-43b1d1-kserve-78fcdfd97c-685tn\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.738109 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.738095 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-model-cache\") pod \"noten-model-43b1d1-kserve-78fcdfd97c-685tn\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.845724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.843659 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-tmp-dir\") pod \"noten-model-43b1d1-kserve-78fcdfd97c-685tn\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.845724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.843736 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjbwk\" (UniqueName: \"kubernetes.io/projected/e4c49e65-ef1f-4f1a-afa9-54d21de21725-kube-api-access-zjbwk\") pod \"noten-model-43b1d1-kserve-78fcdfd97c-685tn\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.845724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.843774 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-dshm\") pod \"noten-model-43b1d1-kserve-78fcdfd97c-685tn\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.845724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.843961 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-home\") pod \"noten-model-43b1d1-kserve-78fcdfd97c-685tn\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.845724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.844036 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-model-cache\") pod \"noten-model-43b1d1-kserve-78fcdfd97c-685tn\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.845724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.844171 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c49e65-ef1f-4f1a-afa9-54d21de21725-tls-certs\") pod \"noten-model-43b1d1-kserve-78fcdfd97c-685tn\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.845724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.845367 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-home\") pod \"noten-model-43b1d1-kserve-78fcdfd97c-685tn\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.845724 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.845595 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-model-cache\") pod \"noten-model-43b1d1-kserve-78fcdfd97c-685tn\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.846374 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.845843 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-tmp-dir\") pod \"noten-model-43b1d1-kserve-78fcdfd97c-685tn\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.863930 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.859484 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c49e65-ef1f-4f1a-afa9-54d21de21725-tls-certs\") pod \"noten-model-43b1d1-kserve-78fcdfd97c-685tn\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.899732 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.899033 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-dshm\") pod \"noten-model-43b1d1-kserve-78fcdfd97c-685tn\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.907029 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.900235 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjbwk\" (UniqueName: \"kubernetes.io/projected/e4c49e65-ef1f-4f1a-afa9-54d21de21725-kube-api-access-zjbwk\") pod \"noten-model-43b1d1-kserve-78fcdfd97c-685tn\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:12.928761 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:12.927687 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:13.293814 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:58:13.293785 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4c49e65_ef1f_4f1a_afa9_54d21de21725.slice/crio-037f11eae60b41e2f938cdca6933901d2e4872a7087b97c58e2f07dd4bc6484f WatchSource:0}: Error finding container 037f11eae60b41e2f938cdca6933901d2e4872a7087b97c58e2f07dd4bc6484f: Status 404 returned error can't find the container with id 037f11eae60b41e2f938cdca6933901d2e4872a7087b97c58e2f07dd4bc6484f Nov 03 16:58:13.294973 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:13.294953 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn"] Nov 03 16:58:13.491894 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:13.491854 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" event={"ID":"e4c49e65-ef1f-4f1a-afa9-54d21de21725","Type":"ContainerStarted","Data":"8d5e22ae1f4806ee65938e8c4d0657ec08090d06c0596bf20455b2940e8f7051"} Nov 03 16:58:13.492113 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:13.491907 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" event={"ID":"e4c49e65-ef1f-4f1a-afa9-54d21de21725","Type":"ContainerStarted","Data":"037f11eae60b41e2f938cdca6933901d2e4872a7087b97c58e2f07dd4bc6484f"} Nov 03 16:58:13.539064 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:13.538999 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" podStartSLOduration=1.5389842969999998 podStartE2EDuration="1.538984297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:58:13.536847506 +0000 UTC m=+2637.125847476" watchObservedRunningTime="2026-11-03 16:58:13.538984297 +0000 UTC m=+2637.127984264" Nov 03 16:58:13.929276 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:13.929247 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:22.929487 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:22.929449 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:22.942230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:22.942207 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:23.582258 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:23.582233 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:27.390871 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:27.390806 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn"] Nov 03 16:58:30.131716 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:58:30.131650 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:58:32.058721 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:32.057822 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:58:42.411955 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.411906 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" podUID="e4c49e65-ef1f-4f1a-afa9-54d21de21725" containerName="main" containerID="cri-o://8d5e22ae1f4806ee65938e8c4d0657ec08090d06c0596bf20455b2940e8f7051" gracePeriod=45 Nov 03 16:58:42.571991 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.571956 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" podUID="e4c49e65-ef1f-4f1a-afa9-54d21de21725" containerName="main" probeResult="failure" output="Get \"https://10.132.0.166:8000/ready\": dial tcp 10.132.0.166:8000: connect: connection refused" Nov 03 16:58:42.689291 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.689265 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:42.736202 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.736174 4947 generic.go:356] "Generic (PLEG): container finished" podID="e4c49e65-ef1f-4f1a-afa9-54d21de21725" containerID="8d5e22ae1f4806ee65938e8c4d0657ec08090d06c0596bf20455b2940e8f7051" exitCode=0 Nov 03 16:58:42.736348 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.736334 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" event={"ID":"e4c49e65-ef1f-4f1a-afa9-54d21de21725","Type":"ContainerDied","Data":"8d5e22ae1f4806ee65938e8c4d0657ec08090d06c0596bf20455b2940e8f7051"} Nov 03 16:58:42.736465 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.736367 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" event={"ID":"e4c49e65-ef1f-4f1a-afa9-54d21de21725","Type":"ContainerDied","Data":"037f11eae60b41e2f938cdca6933901d2e4872a7087b97c58e2f07dd4bc6484f"} Nov 03 16:58:42.736465 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.736383 4947 scope.go:122] "RemoveContainer" containerID="8d5e22ae1f4806ee65938e8c4d0657ec08090d06c0596bf20455b2940e8f7051" Nov 03 16:58:42.736567 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.736492 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn" Nov 03 16:58:42.747185 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.747159 4947 scope.go:122] "RemoveContainer" containerID="8d5e22ae1f4806ee65938e8c4d0657ec08090d06c0596bf20455b2940e8f7051" Nov 03 16:58:42.747482 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:58:42.747456 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d5e22ae1f4806ee65938e8c4d0657ec08090d06c0596bf20455b2940e8f7051\": container with ID starting with 8d5e22ae1f4806ee65938e8c4d0657ec08090d06c0596bf20455b2940e8f7051 not found: ID does not exist" containerID="8d5e22ae1f4806ee65938e8c4d0657ec08090d06c0596bf20455b2940e8f7051" Nov 03 16:58:42.747559 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.747494 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5e22ae1f4806ee65938e8c4d0657ec08090d06c0596bf20455b2940e8f7051"} err="failed to get container status \"8d5e22ae1f4806ee65938e8c4d0657ec08090d06c0596bf20455b2940e8f7051\": rpc error: code = NotFound desc = could not find container \"8d5e22ae1f4806ee65938e8c4d0657ec08090d06c0596bf20455b2940e8f7051\": container with ID starting with 8d5e22ae1f4806ee65938e8c4d0657ec08090d06c0596bf20455b2940e8f7051 not found: ID does not exist" Nov 03 16:58:42.750232 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.750213 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-home\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-home\") pod \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " Nov 03 16:58:42.750336 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.750268 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c49e65-ef1f-4f1a-afa9-54d21de21725-kube-api-access-zjbwk\" (UniqueName: \"kubernetes.io/projected/e4c49e65-ef1f-4f1a-afa9-54d21de21725-kube-api-access-zjbwk\") pod \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " Nov 03 16:58:42.750336 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.750303 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-model-cache\") pod \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " Nov 03 16:58:42.750455 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.750407 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-dshm\") pod \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " Nov 03 16:58:42.750511 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.750474 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-home" pod "e4c49e65-ef1f-4f1a-afa9-54d21de21725" (UID: "e4c49e65-ef1f-4f1a-afa9-54d21de21725"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:42.750571 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.750553 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4c49e65-ef1f-4f1a-afa9-54d21de21725-tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c49e65-ef1f-4f1a-afa9-54d21de21725-tls-certs\") pod \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " Nov 03 16:58:42.750689 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.750670 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-tmp-dir\") pod \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\" (UID: \"e4c49e65-ef1f-4f1a-afa9-54d21de21725\") " Nov 03 16:58:42.750914 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.750889 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-tmp-dir" pod "e4c49e65-ef1f-4f1a-afa9-54d21de21725" (UID: "e4c49e65-ef1f-4f1a-afa9-54d21de21725"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:42.751010 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.750906 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-model-cache" pod "e4c49e65-ef1f-4f1a-afa9-54d21de21725" (UID: "e4c49e65-ef1f-4f1a-afa9-54d21de21725"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:42.751208 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.751187 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:58:42.751441 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.751213 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:58:42.751441 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.751228 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:58:42.753166 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.753120 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-dshm" pod "e4c49e65-ef1f-4f1a-afa9-54d21de21725" (UID: "e4c49e65-ef1f-4f1a-afa9-54d21de21725"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:58:42.753296 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.753247 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c49e65-ef1f-4f1a-afa9-54d21de21725-kube-api-access-zjbwk" pod "e4c49e65-ef1f-4f1a-afa9-54d21de21725" (UID: "e4c49e65-ef1f-4f1a-afa9-54d21de21725"). InnerVolumeSpecName "kube-api-access-zjbwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:58:42.753512 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.753484 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c49e65-ef1f-4f1a-afa9-54d21de21725-tls-certs" pod "e4c49e65-ef1f-4f1a-afa9-54d21de21725" (UID: "e4c49e65-ef1f-4f1a-afa9-54d21de21725"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:58:42.852311 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.852277 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c49e65-ef1f-4f1a-afa9-54d21de21725-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:58:42.852311 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.852305 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjbwk\" (UniqueName: \"kubernetes.io/projected/e4c49e65-ef1f-4f1a-afa9-54d21de21725-kube-api-access-zjbwk\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:58:42.852311 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:42.852315 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c49e65-ef1f-4f1a-afa9-54d21de21725-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:58:43.089963 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:43.089925 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn"] Nov 03 16:58:43.103026 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:43.102996 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-43b1d1/noten-model-43b1d1-kserve-78fcdfd97c-685tn"] Nov 03 16:58:44.566230 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:44.566193 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c49e65-ef1f-4f1a-afa9-54d21de21725" path="/var/lib/kubelet/pods/e4c49e65-ef1f-4f1a-afa9-54d21de21725/volumes" Nov 03 16:58:51.277328 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.277294 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq"] Nov 03 16:58:51.278290 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.278267 4947 cpu_manager.go:414] "containerMap: removing container" podUID="e4c49e65-ef1f-4f1a-afa9-54d21de21725" containerName="main" Nov 03 16:58:51.278290 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.278290 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c49e65-ef1f-4f1a-afa9-54d21de21725" containerName="main" Nov 03 16:58:51.278473 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.278448 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c49e65-ef1f-4f1a-afa9-54d21de21725" containerName="main" Nov 03 16:58:51.282391 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.282371 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.302331 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.302307 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq"] Nov 03 16:58:51.311387 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.311364 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c1837e\"/\"openshift-service-ca.crt\"" Nov 03 16:58:51.311539 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.311376 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c1837e\"/\"autores-model-c1837e-kserve-self-signed-certs\"" Nov 03 16:58:51.311632 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.311364 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c1837e\"/\"kube-root-ca.crt\"" Nov 03 16:58:51.311632 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.311456 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c1837e\"/\"default-dockercfg-fcv4z\"" Nov 03 16:58:51.384879 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.384855 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e8a9a3b-9beb-45b2-8706-b650871abe86-tls-certs\") pod \"autores-model-c1837e-kserve-64c4f54ccc-bf8rq\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.385012 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.384896 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l47hv\" (UniqueName: \"kubernetes.io/projected/6e8a9a3b-9beb-45b2-8706-b650871abe86-kube-api-access-l47hv\") pod \"autores-model-c1837e-kserve-64c4f54ccc-bf8rq\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.385054 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.385004 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-tmp-dir\") pod \"autores-model-c1837e-kserve-64c4f54ccc-bf8rq\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.385135 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.385115 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-dshm\") pod \"autores-model-c1837e-kserve-64c4f54ccc-bf8rq\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.385188 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.385175 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-home\") pod \"autores-model-c1837e-kserve-64c4f54ccc-bf8rq\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.385225 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.385199 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-model-cache\") pod \"autores-model-c1837e-kserve-64c4f54ccc-bf8rq\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.486317 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.486284 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-tmp-dir\") pod \"autores-model-c1837e-kserve-64c4f54ccc-bf8rq\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.486457 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.486386 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-dshm\") pod \"autores-model-c1837e-kserve-64c4f54ccc-bf8rq\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.486457 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.486446 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-home\") pod \"autores-model-c1837e-kserve-64c4f54ccc-bf8rq\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.486574 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.486472 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-model-cache\") pod \"autores-model-c1837e-kserve-64c4f54ccc-bf8rq\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.486630 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.486597 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e8a9a3b-9beb-45b2-8706-b650871abe86-tls-certs\") pod \"autores-model-c1837e-kserve-64c4f54ccc-bf8rq\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.486683 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.486643 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l47hv\" (UniqueName: \"kubernetes.io/projected/6e8a9a3b-9beb-45b2-8706-b650871abe86-kube-api-access-l47hv\") pod \"autores-model-c1837e-kserve-64c4f54ccc-bf8rq\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.486683 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.486647 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-tmp-dir\") pod \"autores-model-c1837e-kserve-64c4f54ccc-bf8rq\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.486833 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.486812 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-home\") pod \"autores-model-c1837e-kserve-64c4f54ccc-bf8rq\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.486903 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.486882 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-model-cache\") pod \"autores-model-c1837e-kserve-64c4f54ccc-bf8rq\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.488690 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.488658 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-dshm\") pod \"autores-model-c1837e-kserve-64c4f54ccc-bf8rq\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.488994 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.488976 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e8a9a3b-9beb-45b2-8706-b650871abe86-tls-certs\") pod \"autores-model-c1837e-kserve-64c4f54ccc-bf8rq\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.543235 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.543143 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l47hv\" (UniqueName: \"kubernetes.io/projected/6e8a9a3b-9beb-45b2-8706-b650871abe86-kube-api-access-l47hv\") pod \"autores-model-c1837e-kserve-64c4f54ccc-bf8rq\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.603461 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.602039 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:58:51.980840 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:58:51.980807 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e8a9a3b_9beb_45b2_8706_b650871abe86.slice/crio-941b5fa5a71731765d3a40b0cc47918fa4632a654de79f7fd867359f1348562f WatchSource:0}: Error finding container 941b5fa5a71731765d3a40b0cc47918fa4632a654de79f7fd867359f1348562f: Status 404 returned error can't find the container with id 941b5fa5a71731765d3a40b0cc47918fa4632a654de79f7fd867359f1348562f Nov 03 16:58:51.981627 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:51.981606 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq"] Nov 03 16:58:52.811083 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:52.811052 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" event={"ID":"6e8a9a3b-9beb-45b2-8706-b650871abe86","Type":"ContainerStarted","Data":"067081916e5d3850389443a2971a8f0ed4b548d694798718db75b2063a5cd4f3"} Nov 03 16:58:52.811083 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:52.811087 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" event={"ID":"6e8a9a3b-9beb-45b2-8706-b650871abe86","Type":"ContainerStarted","Data":"941b5fa5a71731765d3a40b0cc47918fa4632a654de79f7fd867359f1348562f"} Nov 03 16:58:52.865726 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:52.863362 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" podStartSLOduration=1.8633436030000001 podStartE2EDuration="1.863343603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:58:52.857664177 +0000 UTC m=+2676.446664144" watchObservedRunningTime="2026-11-03 16:58:52.863343603 +0000 UTC m=+2676.452343571" Nov 03 16:58:53.604380 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:58:53.604352 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:59:00.130910 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:59:00.130869 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:59:01.604356 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:01.604323 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:59:01.616367 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:01.616345 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:59:01.933085 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:01.933015 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:59:03.922897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:03.922861 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 16:59:16.941541 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:16.941509 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:59:16.944652 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:16.944628 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 16:59:17.892360 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:17.892324 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq"] Nov 03 16:59:19.316044 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:19.316011 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq"] Nov 03 16:59:19.316508 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:19.316242 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" podUID="80a6b2fc-a875-428c-a726-bfd9fc2aaa85" containerName="istio-proxy" containerID="cri-o://75df66069fec869f5fed604bc7d99137a4fcde1214ed51d20f052a1381fba32d" gracePeriod=30 Nov 03 16:59:19.331400 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:19.331370 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8"] Nov 03 16:59:19.331638 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:19.331612 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" podUID="541d8697-b5fa-467e-878c-848106509998" containerName="istio-proxy" containerID="cri-o://de8ccf365c3c707050fd8c976e3fb5c7edf0062b1b6c0a96aa74f2243ddee869" gracePeriod=30 Nov 03 16:59:21.709464 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:21.709431 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp"] Nov 03 16:59:21.715062 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:21.715044 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" Nov 03 16:59:21.738533 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:21.738509 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp"] Nov 03 16:59:21.747738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:21.747717 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-a9426b\"" Nov 03 16:59:21.747738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:21.747726 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-a9426b-dockercfg-cb5nd\"" Nov 03 16:59:21.832727 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:21.815145 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc2a861-7b75-4723-a8b0-269ab7b28ede-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp\" (UID: \"cfc2a861-7b75-4723-a8b0-269ab7b28ede\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" Nov 03 16:59:21.833262 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:21.833218 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfwt4\" (UniqueName: \"kubernetes.io/projected/cfc2a861-7b75-4723-a8b0-269ab7b28ede-kube-api-access-gfwt4\") pod \"payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp\" (UID: \"cfc2a861-7b75-4723-a8b0-269ab7b28ede\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" Nov 03 16:59:21.944988 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:21.941434 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc2a861-7b75-4723-a8b0-269ab7b28ede-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp\" (UID: \"cfc2a861-7b75-4723-a8b0-269ab7b28ede\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" Nov 03 16:59:21.945404 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:21.945374 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfwt4\" (UniqueName: \"kubernetes.io/projected/cfc2a861-7b75-4723-a8b0-269ab7b28ede-kube-api-access-gfwt4\") pod \"payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp\" (UID: \"cfc2a861-7b75-4723-a8b0-269ab7b28ede\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" Nov 03 16:59:21.947044 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:21.946997 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc2a861-7b75-4723-a8b0-269ab7b28ede-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp\" (UID: \"cfc2a861-7b75-4723-a8b0-269ab7b28ede\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" Nov 03 16:59:22.017866 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:22.001766 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx"] Nov 03 16:59:22.017866 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:22.015953 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfwt4\" (UniqueName: \"kubernetes.io/projected/cfc2a861-7b75-4723-a8b0-269ab7b28ede-kube-api-access-gfwt4\") pod \"payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp\" (UID: \"cfc2a861-7b75-4723-a8b0-269ab7b28ede\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" Nov 03 16:59:22.030155 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:22.030109 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" Nov 03 16:59:22.063993 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:22.063968 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" Nov 03 16:59:22.067315 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:22.067118 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx"] Nov 03 16:59:22.156396 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:22.155890 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8b428a-da45-4ad3-ab48-4f2debd415e8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx\" (UID: \"da8b428a-da45-4ad3-ab48-4f2debd415e8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" Nov 03 16:59:22.156396 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:22.155958 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p9n7\" (UniqueName: \"kubernetes.io/projected/da8b428a-da45-4ad3-ab48-4f2debd415e8-kube-api-access-8p9n7\") pod \"payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx\" (UID: \"da8b428a-da45-4ad3-ab48-4f2debd415e8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" Nov 03 16:59:22.257303 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:22.257251 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8b428a-da45-4ad3-ab48-4f2debd415e8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx\" (UID: \"da8b428a-da45-4ad3-ab48-4f2debd415e8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" Nov 03 16:59:22.257609 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:22.257305 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p9n7\" (UniqueName: \"kubernetes.io/projected/da8b428a-da45-4ad3-ab48-4f2debd415e8-kube-api-access-8p9n7\") pod \"payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx\" (UID: \"da8b428a-da45-4ad3-ab48-4f2debd415e8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" Nov 03 16:59:22.258396 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:22.258373 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8b428a-da45-4ad3-ab48-4f2debd415e8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx\" (UID: \"da8b428a-da45-4ad3-ab48-4f2debd415e8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" Nov 03 16:59:22.283007 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:22.282941 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p9n7\" (UniqueName: \"kubernetes.io/projected/da8b428a-da45-4ad3-ab48-4f2debd415e8-kube-api-access-8p9n7\") pod \"payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx\" (UID: \"da8b428a-da45-4ad3-ab48-4f2debd415e8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" Nov 03 16:59:22.379449 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:22.379417 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp"] Nov 03 16:59:22.380819 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:59:22.380795 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc2a861_7b75_4723_a8b0_269ab7b28ede.slice/crio-dca1e00118040293fa3d1d961a9407fb0c576769b452c9a10b3fa131382ef808 WatchSource:0}: Error finding container dca1e00118040293fa3d1d961a9407fb0c576769b452c9a10b3fa131382ef808: Status 404 returned error can't find the container with id dca1e00118040293fa3d1d961a9407fb0c576769b452c9a10b3fa131382ef808 Nov 03 16:59:22.412943 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:22.412911 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" Nov 03 16:59:22.749730 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:22.742518 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx"] Nov 03 16:59:22.763927 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:59:22.763860 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8b428a_da45_4ad3_ab48_4f2debd415e8.slice/crio-1a94b22fc26f5e6a9d2c271a43512f44489d5a098b4ad3f63bfc19e97aba732e WatchSource:0}: Error finding container 1a94b22fc26f5e6a9d2c271a43512f44489d5a098b4ad3f63bfc19e97aba732e: Status 404 returned error can't find the container with id 1a94b22fc26f5e6a9d2c271a43512f44489d5a098b4ad3f63bfc19e97aba732e Nov 03 16:59:23.146031 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:23.145942 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" event={"ID":"cfc2a861-7b75-4723-a8b0-269ab7b28ede","Type":"ContainerStarted","Data":"fa5e85e05517f498468085fcf416733dc130af1962e9a5b183841d0022f28706"} Nov 03 16:59:23.146031 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:23.145998 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" event={"ID":"cfc2a861-7b75-4723-a8b0-269ab7b28ede","Type":"ContainerStarted","Data":"dca1e00118040293fa3d1d961a9407fb0c576769b452c9a10b3fa131382ef808"} Nov 03 16:59:23.146263 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:23.146050 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" Nov 03 16:59:23.147675 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:23.147649 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" event={"ID":"da8b428a-da45-4ad3-ab48-4f2debd415e8","Type":"ContainerStarted","Data":"a031b9a5c71f04507edf6b5150302291b6ed85ebdf952f0ac62ac35f90653e9d"} Nov 03 16:59:23.147818 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:23.147710 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" event={"ID":"da8b428a-da45-4ad3-ab48-4f2debd415e8","Type":"ContainerStarted","Data":"1a94b22fc26f5e6a9d2c271a43512f44489d5a098b4ad3f63bfc19e97aba732e"} Nov 03 16:59:23.147861 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:23.147823 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" Nov 03 16:59:23.207889 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:23.207843 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" podStartSLOduration=2.207829294 podStartE2EDuration="2.207829294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:59:23.20527865 +0000 UTC m=+2706.794278619" watchObservedRunningTime="2026-11-03 16:59:23.207829294 +0000 UTC m=+2706.796829260" Nov 03 16:59:23.273043 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:23.272992 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" podStartSLOduration=2.272978069 podStartE2EDuration="2.272978069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:59:23.269798688 +0000 UTC m=+2706.858798660" watchObservedRunningTime="2026-11-03 16:59:23.272978069 +0000 UTC m=+2706.861978048" Nov 03 16:59:24.718535 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.718508 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:59:24.720522 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.720503 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:59:24.814602 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.814567 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-kube-api-access-7mj2m\" (UniqueName: \"kubernetes.io/projected/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-kube-api-access-7mj2m\") pod \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " Nov 03 16:59:24.814832 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.814659 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/541d8697-b5fa-467e-878c-848106509998-istio-token\" (UniqueName: \"kubernetes.io/projected/541d8697-b5fa-467e-878c-848106509998-istio-token\") pod \"541d8697-b5fa-467e-878c-848106509998\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " Nov 03 16:59:24.814832 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.814794 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istiod-ca-cert\") pod \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " Nov 03 16:59:24.814974 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.814883 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-credential-socket\") pod \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " Nov 03 16:59:24.814974 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.814924 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-workload-socket\") pod \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " Nov 03 16:59:24.815088 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.814978 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-istio-data\") pod \"541d8697-b5fa-467e-878c-848106509998\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " Nov 03 16:59:24.815088 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.815007 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-workload-socket\") pod \"541d8697-b5fa-467e-878c-848106509998\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " Nov 03 16:59:24.815088 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.815058 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-istio-envoy\") pod \"541d8697-b5fa-467e-878c-848106509998\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " Nov 03 16:59:24.815253 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.815103 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-data\") pod \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " Nov 03 16:59:24.815253 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.815129 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-workload-certs\") pod \"541d8697-b5fa-467e-878c-848106509998\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " Nov 03 16:59:24.815253 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.815165 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-credential-socket\") pod \"541d8697-b5fa-467e-878c-848106509998\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " Nov 03 16:59:24.815253 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.815194 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istiod-ca-cert" pod "80a6b2fc-a875-428c-a726-bfd9fc2aaa85" (UID: "80a6b2fc-a875-428c-a726-bfd9fc2aaa85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:59:24.815253 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.815212 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-podinfo\") pod \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " Nov 03 16:59:24.815253 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.815249 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-token\" (UniqueName: \"kubernetes.io/projected/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-token\") pod \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " Nov 03 16:59:24.815569 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.815293 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/541d8697-b5fa-467e-878c-848106509998-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/541d8697-b5fa-467e-878c-848106509998-istiod-ca-cert\") pod \"541d8697-b5fa-467e-878c-848106509998\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " Nov 03 16:59:24.815569 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.815354 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-envoy\") pod \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " Nov 03 16:59:24.815569 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.815431 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/541d8697-b5fa-467e-878c-848106509998-kube-api-access-8wb5r\" (UniqueName: \"kubernetes.io/projected/541d8697-b5fa-467e-878c-848106509998-kube-api-access-8wb5r\") pod \"541d8697-b5fa-467e-878c-848106509998\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " Nov 03 16:59:24.815569 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.815464 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/541d8697-b5fa-467e-878c-848106509998-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/541d8697-b5fa-467e-878c-848106509998-istio-podinfo\") pod \"541d8697-b5fa-467e-878c-848106509998\" (UID: \"541d8697-b5fa-467e-878c-848106509998\") " Nov 03 16:59:24.815806 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.815598 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-workload-certs\") pod \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\" (UID: \"80a6b2fc-a875-428c-a726-bfd9fc2aaa85\") " Nov 03 16:59:24.815806 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.815604 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-credential-socket" pod "80a6b2fc-a875-428c-a726-bfd9fc2aaa85" (UID: "80a6b2fc-a875-428c-a726-bfd9fc2aaa85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:24.816413 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.816373 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:24.816413 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.816403 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:24.816638 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.816615 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-workload-certs" pod "80a6b2fc-a875-428c-a726-bfd9fc2aaa85" (UID: "80a6b2fc-a875-428c-a726-bfd9fc2aaa85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:24.817899 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.817865 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-kube-api-access-7mj2m" pod "80a6b2fc-a875-428c-a726-bfd9fc2aaa85" (UID: "80a6b2fc-a875-428c-a726-bfd9fc2aaa85"). InnerVolumeSpecName "kube-api-access-7mj2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:24.818315 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.818294 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-envoy" pod "80a6b2fc-a875-428c-a726-bfd9fc2aaa85" (UID: "80a6b2fc-a875-428c-a726-bfd9fc2aaa85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:24.818621 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.818601 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-token" pod "80a6b2fc-a875-428c-a726-bfd9fc2aaa85" (UID: "80a6b2fc-a875-428c-a726-bfd9fc2aaa85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:24.820068 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.820043 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-istio-data" pod "541d8697-b5fa-467e-878c-848106509998" (UID: "541d8697-b5fa-467e-878c-848106509998"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:24.820169 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.820108 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/541d8697-b5fa-467e-878c-848106509998-istiod-ca-cert" pod "541d8697-b5fa-467e-878c-848106509998" (UID: "541d8697-b5fa-467e-878c-848106509998"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:59:24.820869 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.820798 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-podinfo" pod "80a6b2fc-a875-428c-a726-bfd9fc2aaa85" (UID: "80a6b2fc-a875-428c-a726-bfd9fc2aaa85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:59:24.820964 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.820917 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-workload-socket" pod "80a6b2fc-a875-428c-a726-bfd9fc2aaa85" (UID: "80a6b2fc-a875-428c-a726-bfd9fc2aaa85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:24.820964 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.820931 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-workload-certs" pod "541d8697-b5fa-467e-878c-848106509998" (UID: "541d8697-b5fa-467e-878c-848106509998"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:24.821101 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.820962 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-credential-socket" pod "541d8697-b5fa-467e-878c-848106509998" (UID: "541d8697-b5fa-467e-878c-848106509998"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:24.821380 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.821183 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-data" pod "80a6b2fc-a875-428c-a726-bfd9fc2aaa85" (UID: "80a6b2fc-a875-428c-a726-bfd9fc2aaa85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:24.825261 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.825240 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/541d8697-b5fa-467e-878c-848106509998-istio-token" pod "541d8697-b5fa-467e-878c-848106509998" (UID: "541d8697-b5fa-467e-878c-848106509998"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:24.825261 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.825251 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/541d8697-b5fa-467e-878c-848106509998-istio-podinfo" pod "541d8697-b5fa-467e-878c-848106509998" (UID: "541d8697-b5fa-467e-878c-848106509998"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 16:59:24.826087 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.826052 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-istio-envoy" pod "541d8697-b5fa-467e-878c-848106509998" (UID: "541d8697-b5fa-467e-878c-848106509998"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:24.826822 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.826790 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-workload-socket" pod "541d8697-b5fa-467e-878c-848106509998" (UID: "541d8697-b5fa-467e-878c-848106509998"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:24.827043 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.827018 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/541d8697-b5fa-467e-878c-848106509998-kube-api-access-8wb5r" pod "541d8697-b5fa-467e-878c-848106509998" (UID: "541d8697-b5fa-467e-878c-848106509998"). InnerVolumeSpecName "kube-api-access-8wb5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:24.919776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.919629 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:24.919776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.919667 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wb5r\" (UniqueName: \"kubernetes.io/projected/541d8697-b5fa-467e-878c-848106509998-kube-api-access-8wb5r\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:24.919776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.919682 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/541d8697-b5fa-467e-878c-848106509998-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:24.919776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.919711 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:24.919776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.919724 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mj2m\" (UniqueName: \"kubernetes.io/projected/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-kube-api-access-7mj2m\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:24.919776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.919736 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/541d8697-b5fa-467e-878c-848106509998-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:24.919776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.919753 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:24.919776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.919766 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:24.919776 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.919779 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:24.920285 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.919791 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:24.920285 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.919803 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:24.920285 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.919817 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:24.920285 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.919828 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/541d8697-b5fa-467e-878c-848106509998-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:24.920285 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.919840 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:24.920285 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.919852 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80a6b2fc-a875-428c-a726-bfd9fc2aaa85-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:24.920285 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:24.919867 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/541d8697-b5fa-467e-878c-848106509998-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:25.161907 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:25.161876 4947 generic.go:356] "Generic (PLEG): container finished" podID="80a6b2fc-a875-428c-a726-bfd9fc2aaa85" containerID="75df66069fec869f5fed604bc7d99137a4fcde1214ed51d20f052a1381fba32d" exitCode=0 Nov 03 16:59:25.162072 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:25.161916 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" event={"ID":"80a6b2fc-a875-428c-a726-bfd9fc2aaa85","Type":"ContainerDied","Data":"75df66069fec869f5fed604bc7d99137a4fcde1214ed51d20f052a1381fba32d"} Nov 03 16:59:25.162072 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:25.161987 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" event={"ID":"80a6b2fc-a875-428c-a726-bfd9fc2aaa85","Type":"ContainerDied","Data":"28e32dd5e5b19510d8079303abcbda944fe9c8c2340decf4a430094b544f6366"} Nov 03 16:59:25.162072 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:25.161930 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq" Nov 03 16:59:25.162072 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:25.162017 4947 scope.go:122] "RemoveContainer" containerID="75df66069fec869f5fed604bc7d99137a4fcde1214ed51d20f052a1381fba32d" Nov 03 16:59:25.163424 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:25.163399 4947 generic.go:356] "Generic (PLEG): container finished" podID="541d8697-b5fa-467e-878c-848106509998" containerID="de8ccf365c3c707050fd8c976e3fb5c7edf0062b1b6c0a96aa74f2243ddee869" exitCode=0 Nov 03 16:59:25.163738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:25.163437 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" Nov 03 16:59:25.163738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:25.163483 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" event={"ID":"541d8697-b5fa-467e-878c-848106509998","Type":"ContainerDied","Data":"de8ccf365c3c707050fd8c976e3fb5c7edf0062b1b6c0a96aa74f2243ddee869"} Nov 03 16:59:25.163738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:25.163534 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8" event={"ID":"541d8697-b5fa-467e-878c-848106509998","Type":"ContainerDied","Data":"f162f847eb7085183c61f3e3d78e03f5d2dac0f1215485019a1708ad13131ef3"} Nov 03 16:59:25.173476 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:25.173455 4947 scope.go:122] "RemoveContainer" containerID="75df66069fec869f5fed604bc7d99137a4fcde1214ed51d20f052a1381fba32d" Nov 03 16:59:25.173801 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:59:25.173775 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75df66069fec869f5fed604bc7d99137a4fcde1214ed51d20f052a1381fba32d\": container with ID starting with 75df66069fec869f5fed604bc7d99137a4fcde1214ed51d20f052a1381fba32d not found: ID does not exist" containerID="75df66069fec869f5fed604bc7d99137a4fcde1214ed51d20f052a1381fba32d" Nov 03 16:59:25.173935 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:25.173833 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75df66069fec869f5fed604bc7d99137a4fcde1214ed51d20f052a1381fba32d"} err="failed to get container status \"75df66069fec869f5fed604bc7d99137a4fcde1214ed51d20f052a1381fba32d\": rpc error: code = NotFound desc = could not find container \"75df66069fec869f5fed604bc7d99137a4fcde1214ed51d20f052a1381fba32d\": container with ID starting with 75df66069fec869f5fed604bc7d99137a4fcde1214ed51d20f052a1381fba32d not found: ID does not exist" Nov 03 16:59:25.173935 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:25.173858 4947 scope.go:122] "RemoveContainer" containerID="de8ccf365c3c707050fd8c976e3fb5c7edf0062b1b6c0a96aa74f2243ddee869" Nov 03 16:59:25.185027 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:25.185007 4947 scope.go:122] "RemoveContainer" containerID="de8ccf365c3c707050fd8c976e3fb5c7edf0062b1b6c0a96aa74f2243ddee869" Nov 03 16:59:25.185317 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:59:25.185300 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de8ccf365c3c707050fd8c976e3fb5c7edf0062b1b6c0a96aa74f2243ddee869\": container with ID starting with de8ccf365c3c707050fd8c976e3fb5c7edf0062b1b6c0a96aa74f2243ddee869 not found: ID does not exist" containerID="de8ccf365c3c707050fd8c976e3fb5c7edf0062b1b6c0a96aa74f2243ddee869" Nov 03 16:59:25.185365 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:25.185324 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de8ccf365c3c707050fd8c976e3fb5c7edf0062b1b6c0a96aa74f2243ddee869"} err="failed to get container status \"de8ccf365c3c707050fd8c976e3fb5c7edf0062b1b6c0a96aa74f2243ddee869\": rpc error: code = NotFound desc = could not find container \"de8ccf365c3c707050fd8c976e3fb5c7edf0062b1b6c0a96aa74f2243ddee869\": container with ID starting with de8ccf365c3c707050fd8c976e3fb5c7edf0062b1b6c0a96aa74f2243ddee869 not found: ID does not exist" Nov 03 16:59:25.220942 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:25.220913 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8"] Nov 03 16:59:25.234016 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:25.233991 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-kq5w8"] Nov 03 16:59:25.278771 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:25.278744 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq"] Nov 03 16:59:25.285279 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:25.285253 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-9xzzq"] Nov 03 16:59:26.566921 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:26.566891 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="541d8697-b5fa-467e-878c-848106509998" path="/var/lib/kubelet/pods/541d8697-b5fa-467e-878c-848106509998/volumes" Nov 03 16:59:26.567452 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:26.567437 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80a6b2fc-a875-428c-a726-bfd9fc2aaa85" path="/var/lib/kubelet/pods/80a6b2fc-a875-428c-a726-bfd9fc2aaa85/volumes" Nov 03 16:59:30.132855 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:59:30.132822 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 16:59:32.942660 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:32.942619 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" podUID="6e8a9a3b-9beb-45b2-8706-b650871abe86" containerName="main" containerID="cri-o://067081916e5d3850389443a2971a8f0ed4b548d694798718db75b2063a5cd4f3" gracePeriod=45 Nov 03 16:59:33.272877 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.272824 4947 generic.go:356] "Generic (PLEG): container finished" podID="6e8a9a3b-9beb-45b2-8706-b650871abe86" containerID="067081916e5d3850389443a2971a8f0ed4b548d694798718db75b2063a5cd4f3" exitCode=0 Nov 03 16:59:33.273006 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.272899 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" event={"ID":"6e8a9a3b-9beb-45b2-8706-b650871abe86","Type":"ContainerDied","Data":"067081916e5d3850389443a2971a8f0ed4b548d694798718db75b2063a5cd4f3"} Nov 03 16:59:33.292535 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.292506 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:59:33.401861 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.401821 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e8a9a3b-9beb-45b2-8706-b650871abe86-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e8a9a3b-9beb-45b2-8706-b650871abe86-tls-certs\") pod \"6e8a9a3b-9beb-45b2-8706-b650871abe86\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " Nov 03 16:59:33.401861 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.401867 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-model-cache\") pod \"6e8a9a3b-9beb-45b2-8706-b650871abe86\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " Nov 03 16:59:33.402082 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.401903 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-home\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-home\") pod \"6e8a9a3b-9beb-45b2-8706-b650871abe86\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " Nov 03 16:59:33.402082 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.401993 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e8a9a3b-9beb-45b2-8706-b650871abe86-kube-api-access-l47hv\" (UniqueName: \"kubernetes.io/projected/6e8a9a3b-9beb-45b2-8706-b650871abe86-kube-api-access-l47hv\") pod \"6e8a9a3b-9beb-45b2-8706-b650871abe86\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " Nov 03 16:59:33.402172 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.402093 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-dshm\") pod \"6e8a9a3b-9beb-45b2-8706-b650871abe86\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " Nov 03 16:59:33.402172 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.402159 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-tmp-dir\") pod \"6e8a9a3b-9beb-45b2-8706-b650871abe86\" (UID: \"6e8a9a3b-9beb-45b2-8706-b650871abe86\") " Nov 03 16:59:33.402262 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.402176 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-model-cache" pod "6e8a9a3b-9beb-45b2-8706-b650871abe86" (UID: "6e8a9a3b-9beb-45b2-8706-b650871abe86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:33.402262 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.402212 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-home" pod "6e8a9a3b-9beb-45b2-8706-b650871abe86" (UID: "6e8a9a3b-9beb-45b2-8706-b650871abe86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:33.402480 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.402455 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-tmp-dir" pod "6e8a9a3b-9beb-45b2-8706-b650871abe86" (UID: "6e8a9a3b-9beb-45b2-8706-b650871abe86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:33.402686 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.402669 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:33.402765 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.402691 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:33.402765 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.402719 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:33.404407 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.404383 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8a9a3b-9beb-45b2-8706-b650871abe86-kube-api-access-l47hv" pod "6e8a9a3b-9beb-45b2-8706-b650871abe86" (UID: "6e8a9a3b-9beb-45b2-8706-b650871abe86"). InnerVolumeSpecName "kube-api-access-l47hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:33.404738 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.404716 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-dshm" pod "6e8a9a3b-9beb-45b2-8706-b650871abe86" (UID: "6e8a9a3b-9beb-45b2-8706-b650871abe86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 16:59:33.404793 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.404692 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e8a9a3b-9beb-45b2-8706-b650871abe86-tls-certs" pod "6e8a9a3b-9beb-45b2-8706-b650871abe86" (UID: "6e8a9a3b-9beb-45b2-8706-b650871abe86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 16:59:33.503752 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.503689 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e8a9a3b-9beb-45b2-8706-b650871abe86-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:33.503752 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.503749 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l47hv\" (UniqueName: \"kubernetes.io/projected/6e8a9a3b-9beb-45b2-8706-b650871abe86-kube-api-access-l47hv\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:33.503752 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:33.503758 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e8a9a3b-9beb-45b2-8706-b650871abe86-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:34.158098 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:34.158069 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" Nov 03 16:59:34.158458 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:34.158151 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" Nov 03 16:59:34.463937 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:34.463851 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" event={"ID":"6e8a9a3b-9beb-45b2-8706-b650871abe86","Type":"ContainerDied","Data":"941b5fa5a71731765d3a40b0cc47918fa4632a654de79f7fd867359f1348562f"} Nov 03 16:59:34.464280 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:34.464263 4947 scope.go:122] "RemoveContainer" containerID="067081916e5d3850389443a2971a8f0ed4b548d694798718db75b2063a5cd4f3" Nov 03 16:59:34.464533 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:34.464202 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq" Nov 03 16:59:34.566897 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:34.566868 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq"] Nov 03 16:59:34.591445 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:34.591417 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-c1837e/autores-model-c1837e-kserve-64c4f54ccc-bf8rq"] Nov 03 16:59:36.567561 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:36.567518 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e8a9a3b-9beb-45b2-8706-b650871abe86" path="/var/lib/kubelet/pods/6e8a9a3b-9beb-45b2-8706-b650871abe86/volumes" Nov 03 16:59:40.066929 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:40.066893 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq"] Nov 03 16:59:45.067600 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.067557 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq" podUID="1ae217bd-e9a6-497c-bc9a-54cfc421db75" containerName="payload-pre-processing" containerID="cri-o://ffef7990826c855bcb3b15429a4645dec9430c4494bd3d86c8255d10624d247e" gracePeriod=25 Nov 03 16:59:45.250635 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.250589 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg"] Nov 03 16:59:45.251294 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.251279 4947 cpu_manager.go:414] "containerMap: removing container" podUID="541d8697-b5fa-467e-878c-848106509998" containerName="istio-proxy" Nov 03 16:59:45.251347 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.251296 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541d8697-b5fa-467e-878c-848106509998" containerName="istio-proxy" Nov 03 16:59:45.251347 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.251317 4947 cpu_manager.go:414] "containerMap: removing container" podUID="6e8a9a3b-9beb-45b2-8706-b650871abe86" containerName="main" Nov 03 16:59:45.251347 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.251322 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e8a9a3b-9beb-45b2-8706-b650871abe86" containerName="main" Nov 03 16:59:45.251443 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.251354 4947 cpu_manager.go:414] "containerMap: removing container" podUID="80a6b2fc-a875-428c-a726-bfd9fc2aaa85" containerName="istio-proxy" Nov 03 16:59:45.251443 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.251360 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a6b2fc-a875-428c-a726-bfd9fc2aaa85" containerName="istio-proxy" Nov 03 16:59:45.251501 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.251471 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a6b2fc-a875-428c-a726-bfd9fc2aaa85" containerName="istio-proxy" Nov 03 16:59:45.251501 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.251483 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="541d8697-b5fa-467e-878c-848106509998" containerName="istio-proxy" Nov 03 16:59:45.251501 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.251490 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e8a9a3b-9beb-45b2-8706-b650871abe86" containerName="main" Nov 03 16:59:45.255034 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.255013 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" Nov 03 16:59:45.277749 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.277687 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-a9426b-serving-cert\"" Nov 03 16:59:45.277896 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.277840 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg"] Nov 03 16:59:45.332146 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.332062 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6739e09f-4597-4915-8400-ecc28a6e122c-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg\" (UID: \"6739e09f-4597-4915-8400-ecc28a6e122c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" Nov 03 16:59:45.332307 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.332184 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb2vr\" (UniqueName: \"kubernetes.io/projected/6739e09f-4597-4915-8400-ecc28a6e122c-kube-api-access-fb2vr\") pod \"maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg\" (UID: \"6739e09f-4597-4915-8400-ecc28a6e122c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" Nov 03 16:59:45.332307 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.332278 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6739e09f-4597-4915-8400-ecc28a6e122c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg\" (UID: \"6739e09f-4597-4915-8400-ecc28a6e122c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" Nov 03 16:59:45.332421 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.332361 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6739e09f-4597-4915-8400-ecc28a6e122c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg\" (UID: \"6739e09f-4597-4915-8400-ecc28a6e122c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" Nov 03 16:59:45.389667 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.389643 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq" Nov 03 16:59:45.434687 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.434647 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6739e09f-4597-4915-8400-ecc28a6e122c-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg\" (UID: \"6739e09f-4597-4915-8400-ecc28a6e122c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" Nov 03 16:59:45.434940 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.434738 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb2vr\" (UniqueName: \"kubernetes.io/projected/6739e09f-4597-4915-8400-ecc28a6e122c-kube-api-access-fb2vr\") pod \"maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg\" (UID: \"6739e09f-4597-4915-8400-ecc28a6e122c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" Nov 03 16:59:45.434940 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.434811 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6739e09f-4597-4915-8400-ecc28a6e122c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg\" (UID: \"6739e09f-4597-4915-8400-ecc28a6e122c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" Nov 03 16:59:45.434940 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.434842 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6739e09f-4597-4915-8400-ecc28a6e122c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg\" (UID: \"6739e09f-4597-4915-8400-ecc28a6e122c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" Nov 03 16:59:45.436487 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.436458 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6739e09f-4597-4915-8400-ecc28a6e122c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg\" (UID: \"6739e09f-4597-4915-8400-ecc28a6e122c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" Nov 03 16:59:45.438820 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.438796 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6739e09f-4597-4915-8400-ecc28a6e122c-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg\" (UID: \"6739e09f-4597-4915-8400-ecc28a6e122c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" Nov 03 16:59:45.439215 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.439177 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6739e09f-4597-4915-8400-ecc28a6e122c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg\" (UID: \"6739e09f-4597-4915-8400-ecc28a6e122c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" Nov 03 16:59:45.461210 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.461184 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb2vr\" (UniqueName: \"kubernetes.io/projected/6739e09f-4597-4915-8400-ecc28a6e122c-kube-api-access-fb2vr\") pod \"maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg\" (UID: \"6739e09f-4597-4915-8400-ecc28a6e122c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" Nov 03 16:59:45.550210 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.550172 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae217bd-e9a6-497c-bc9a-54cfc421db75-kube-api-access-n7thm\" (UniqueName: \"kubernetes.io/projected/1ae217bd-e9a6-497c-bc9a-54cfc421db75-kube-api-access-n7thm\") pod \"1ae217bd-e9a6-497c-bc9a-54cfc421db75\" (UID: \"1ae217bd-e9a6-497c-bc9a-54cfc421db75\") " Nov 03 16:59:45.567729 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.563866 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae217bd-e9a6-497c-bc9a-54cfc421db75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae217bd-e9a6-497c-bc9a-54cfc421db75-plugins-config-volume\") pod \"1ae217bd-e9a6-497c-bc9a-54cfc421db75\" (UID: \"1ae217bd-e9a6-497c-bc9a-54cfc421db75\") " Nov 03 16:59:45.567729 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.557206 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq" event={"ID":"1ae217bd-e9a6-497c-bc9a-54cfc421db75","Type":"ContainerDied","Data":"ffef7990826c855bcb3b15429a4645dec9430c4494bd3d86c8255d10624d247e"} Nov 03 16:59:45.588835 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.557177 4947 generic.go:356] "Generic (PLEG): container finished" podID="1ae217bd-e9a6-497c-bc9a-54cfc421db75" containerID="ffef7990826c855bcb3b15429a4645dec9430c4494bd3d86c8255d10624d247e" exitCode=0 Nov 03 16:59:45.588835 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.573524 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq" event={"ID":"1ae217bd-e9a6-497c-bc9a-54cfc421db75","Type":"ContainerDied","Data":"7e6bcfb651e8d2e4b8084fe25c9a177f7071452910eb04e676c4a4e7cd186a10"} Nov 03 16:59:45.588835 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.573966 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae217bd-e9a6-497c-bc9a-54cfc421db75-plugins-config-volume" pod "1ae217bd-e9a6-497c-bc9a-54cfc421db75" (UID: "1ae217bd-e9a6-497c-bc9a-54cfc421db75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 16:59:45.588835 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.557270 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq" Nov 03 16:59:45.588835 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.574490 4947 scope.go:122] "RemoveContainer" containerID="ffef7990826c855bcb3b15429a4645dec9430c4494bd3d86c8255d10624d247e" Nov 03 16:59:45.588835 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.574988 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" Nov 03 16:59:45.588835 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.576185 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae217bd-e9a6-497c-bc9a-54cfc421db75-kube-api-access-n7thm" pod "1ae217bd-e9a6-497c-bc9a-54cfc421db75" (UID: "1ae217bd-e9a6-497c-bc9a-54cfc421db75"). InnerVolumeSpecName "kube-api-access-n7thm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:45.619121 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.619095 4947 scope.go:122] "RemoveContainer" containerID="ffef7990826c855bcb3b15429a4645dec9430c4494bd3d86c8255d10624d247e" Nov 03 16:59:45.619519 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:59:45.619448 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffef7990826c855bcb3b15429a4645dec9430c4494bd3d86c8255d10624d247e\": container with ID starting with ffef7990826c855bcb3b15429a4645dec9430c4494bd3d86c8255d10624d247e not found: ID does not exist" containerID="ffef7990826c855bcb3b15429a4645dec9430c4494bd3d86c8255d10624d247e" Nov 03 16:59:45.619519 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.619491 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffef7990826c855bcb3b15429a4645dec9430c4494bd3d86c8255d10624d247e"} err="failed to get container status \"ffef7990826c855bcb3b15429a4645dec9430c4494bd3d86c8255d10624d247e\": rpc error: code = NotFound desc = could not find container \"ffef7990826c855bcb3b15429a4645dec9430c4494bd3d86c8255d10624d247e\": container with ID starting with ffef7990826c855bcb3b15429a4645dec9430c4494bd3d86c8255d10624d247e not found: ID does not exist" Nov 03 16:59:45.673894 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.673839 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7thm\" (UniqueName: \"kubernetes.io/projected/1ae217bd-e9a6-497c-bc9a-54cfc421db75-kube-api-access-n7thm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:45.673894 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.673871 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae217bd-e9a6-497c-bc9a-54cfc421db75-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:45.933420 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:59:45.933392 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6739e09f_4597_4915_8400_ecc28a6e122c.slice/crio-801d361c20005fd3ac5b9c46bfcacbf7c7d99ab6d78ba81bd69a36a7fa95ac9f WatchSource:0}: Error finding container 801d361c20005fd3ac5b9c46bfcacbf7c7d99ab6d78ba81bd69a36a7fa95ac9f: Status 404 returned error can't find the container with id 801d361c20005fd3ac5b9c46bfcacbf7c7d99ab6d78ba81bd69a36a7fa95ac9f Nov 03 16:59:45.935153 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.935131 4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 16:59:45.938020 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.937998 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg"] Nov 03 16:59:45.941811 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.941787 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq"] Nov 03 16:59:45.953194 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:45.953171 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-c1837e-68dffb48ff-j5ndq"] Nov 03 16:59:46.663029 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:46.662946 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae217bd-e9a6-497c-bc9a-54cfc421db75" path="/var/lib/kubelet/pods/1ae217bd-e9a6-497c-bc9a-54cfc421db75/volumes" Nov 03 16:59:46.663423 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:46.663398 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" Nov 03 16:59:46.663548 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:46.663447 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" event={"ID":"6739e09f-4597-4915-8400-ecc28a6e122c","Type":"ContainerStarted","Data":"3fc5418a8a13253603f5be22716054660f6ead2d05f0c54da321cf32c29ae457"} Nov 03 16:59:46.663548 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:46.663475 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" event={"ID":"6739e09f-4597-4915-8400-ecc28a6e122c","Type":"ContainerStarted","Data":"801d361c20005fd3ac5b9c46bfcacbf7c7d99ab6d78ba81bd69a36a7fa95ac9f"} Nov 03 16:59:46.718662 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:46.718601 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" podStartSLOduration=1.3708347 podStartE2EDuration="1.718584024s" totalImagesPullingTime="347.749324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:59:46.717609185 +0000 UTC m=+2730.306609153" watchObservedRunningTime="2026-11-03 16:59:46.718584024 +0000 UTC m=+2730.307583993" Nov 03 16:59:49.900217 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:49.900185 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11348-dc6313"] Nov 03 16:59:49.900881 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:49.900862 4947 cpu_manager.go:414] "containerMap: removing container" podUID="1ae217bd-e9a6-497c-bc9a-54cfc421db75" containerName="payload-pre-processing" Nov 03 16:59:49.900997 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:49.900882 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae217bd-e9a6-497c-bc9a-54cfc421db75" containerName="payload-pre-processing" Nov 03 16:59:49.901070 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:49.901049 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae217bd-e9a6-497c-bc9a-54cfc421db75" containerName="payload-pre-processing" Nov 03 16:59:49.905719 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:49.905684 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11348-dc6313" Nov 03 16:59:49.923987 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:49.923965 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11348-dc6313"] Nov 03 16:59:49.932572 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:49.932539 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-z6dp7\"" Nov 03 16:59:50.025260 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:50.025228 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-654st\" (UniqueName: \"kubernetes.io/projected/4d75eed1-8b6d-4e3c-af3c-57516addc1f1-kube-api-access-654st\") pod \"test-curl-11348-dc6313\" (UID: \"4d75eed1-8b6d-4e3c-af3c-57516addc1f1\") " pod="opendatahub/test-curl-11348-dc6313" Nov 03 16:59:50.126613 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:50.126580 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-654st\" (UniqueName: \"kubernetes.io/projected/4d75eed1-8b6d-4e3c-af3c-57516addc1f1-kube-api-access-654st\") pod \"test-curl-11348-dc6313\" (UID: \"4d75eed1-8b6d-4e3c-af3c-57516addc1f1\") " pod="opendatahub/test-curl-11348-dc6313" Nov 03 16:59:50.151352 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:50.151285 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-654st\" (UniqueName: \"kubernetes.io/projected/4d75eed1-8b6d-4e3c-af3c-57516addc1f1-kube-api-access-654st\") pod \"test-curl-11348-dc6313\" (UID: \"4d75eed1-8b6d-4e3c-af3c-57516addc1f1\") " pod="opendatahub/test-curl-11348-dc6313" Nov 03 16:59:50.219401 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:50.219371 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11348-dc6313" Nov 03 16:59:50.525001 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:59:50.524968 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d75eed1_8b6d_4e3c_af3c_57516addc1f1.slice/crio-09eb80499be2310499874fef5662bf54a67b81d7ec299925339a6a961d43467a WatchSource:0}: Error finding container 09eb80499be2310499874fef5662bf54a67b81d7ec299925339a6a961d43467a: Status 404 returned error can't find the container with id 09eb80499be2310499874fef5662bf54a67b81d7ec299925339a6a961d43467a Nov 03 16:59:50.525501 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:50.525461 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11348-dc6313"] Nov 03 16:59:50.693523 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:50.693487 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11348-dc6313" event={"ID":"4d75eed1-8b6d-4e3c-af3c-57516addc1f1","Type":"ContainerStarted","Data":"09eb80499be2310499874fef5662bf54a67b81d7ec299925339a6a961d43467a"} Nov 03 16:59:52.676252 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:52.676227 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" Nov 03 16:59:52.723900 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:52.723862 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11348-dc6313" event={"ID":"4d75eed1-8b6d-4e3c-af3c-57516addc1f1","Type":"ContainerStarted","Data":"717c3a91f2df2abc09d643cd0c97a56898c98cf8c4434b6af1068f99efbfdcab"} Nov 03 16:59:52.781659 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:52.781581 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11348-dc6313" podStartSLOduration=1.7575895689999999 podStartE2EDuration="3.781562666s" totalImagesPullingTime="2.023973097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:59:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:59:52.781344028 +0000 UTC m=+2736.370343996" watchObservedRunningTime="2026-11-03 16:59:52.781562666 +0000 UTC m=+2736.370562635" Nov 03 16:59:53.738975 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:53.738945 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11348-dc6313"] Nov 03 16:59:53.744093 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:53.744069 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11348-dc6313"] Nov 03 16:59:53.922521 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:53.922489 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11348-4c1fae"] Nov 03 16:59:53.923471 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:53.923451 4947 cpu_manager.go:414] "containerMap: removing container" podUID="4d75eed1-8b6d-4e3c-af3c-57516addc1f1" containerName="test-curl-11348-dc6313" Nov 03 16:59:53.923549 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:53.923472 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d75eed1-8b6d-4e3c-af3c-57516addc1f1" containerName="test-curl-11348-dc6313" Nov 03 16:59:53.923656 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:53.923644 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d75eed1-8b6d-4e3c-af3c-57516addc1f1" containerName="test-curl-11348-dc6313" Nov 03 16:59:53.927960 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:53.927941 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11348-4c1fae" Nov 03 16:59:53.943484 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:53.943455 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11348-4c1fae"] Nov 03 16:59:54.017214 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:54.017186 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp9mm\" (UniqueName: \"kubernetes.io/projected/eaf834a9-d570-42dc-b01f-3a7565bf187d-kube-api-access-jp9mm\") pod \"test-curl-11348-4c1fae\" (UID: \"eaf834a9-d570-42dc-b01f-3a7565bf187d\") " pod="opendatahub/test-curl-11348-4c1fae" Nov 03 16:59:54.119090 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:54.119061 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp9mm\" (UniqueName: \"kubernetes.io/projected/eaf834a9-d570-42dc-b01f-3a7565bf187d-kube-api-access-jp9mm\") pod \"test-curl-11348-4c1fae\" (UID: \"eaf834a9-d570-42dc-b01f-3a7565bf187d\") " pod="opendatahub/test-curl-11348-4c1fae" Nov 03 16:59:54.150216 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:54.150184 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp9mm\" (UniqueName: \"kubernetes.io/projected/eaf834a9-d570-42dc-b01f-3a7565bf187d-kube-api-access-jp9mm\") pod \"test-curl-11348-4c1fae\" (UID: \"eaf834a9-d570-42dc-b01f-3a7565bf187d\") " pod="opendatahub/test-curl-11348-4c1fae" Nov 03 16:59:54.241178 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:54.241149 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11348-4c1fae" Nov 03 16:59:54.557610 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:54.557577 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11348-4c1fae"] Nov 03 16:59:54.558762 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:59:54.558615 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeaf834a9_d570_42dc_b01f_3a7565bf187d.slice/crio-04a76bd6eeb970b5d08cfde8d372ad817f0d0ac89f1a172b391df16d91434288 WatchSource:0}: Error finding container 04a76bd6eeb970b5d08cfde8d372ad817f0d0ac89f1a172b391df16d91434288: Status 404 returned error can't find the container with id 04a76bd6eeb970b5d08cfde8d372ad817f0d0ac89f1a172b391df16d91434288 Nov 03 16:59:54.789838 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:54.788421 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11348-dc6313" podUID="4d75eed1-8b6d-4e3c-af3c-57516addc1f1" containerName="test-curl-11348-dc6313" containerID="cri-o://717c3a91f2df2abc09d643cd0c97a56898c98cf8c4434b6af1068f99efbfdcab" gracePeriod=2 Nov 03 16:59:54.789838 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:54.789552 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11348-4c1fae" event={"ID":"eaf834a9-d570-42dc-b01f-3a7565bf187d","Type":"ContainerStarted","Data":"758f17e83dc5d2406bd5fed8ea0682faad7563cc5a703145136c01854590d05c"} Nov 03 16:59:54.789838 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:54.789596 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11348-4c1fae" event={"ID":"eaf834a9-d570-42dc-b01f-3a7565bf187d","Type":"ContainerStarted","Data":"04a76bd6eeb970b5d08cfde8d372ad817f0d0ac89f1a172b391df16d91434288"} Nov 03 16:59:54.853829 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:54.853731 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11348-4c1fae" podStartSLOduration=1.853691398 podStartE2EDuration="1.853691398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:59:54.85323805 +0000 UTC m=+2738.442238017" watchObservedRunningTime="2026-11-03 16:59:54.853691398 +0000 UTC m=+2738.442691366" Nov 03 16:59:55.969275 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:55.969239 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11348-4c1fae"] Nov 03 16:59:55.978152 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:55.978126 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11348-4c1fae"] Nov 03 16:59:56.111750 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:56.111722 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11348-c85649"] Nov 03 16:59:56.112390 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:56.112376 4947 cpu_manager.go:414] "containerMap: removing container" podUID="eaf834a9-d570-42dc-b01f-3a7565bf187d" containerName="test-curl-11348-4c1fae" Nov 03 16:59:56.112450 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:56.112391 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf834a9-d570-42dc-b01f-3a7565bf187d" containerName="test-curl-11348-4c1fae" Nov 03 16:59:56.112495 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:56.112486 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf834a9-d570-42dc-b01f-3a7565bf187d" containerName="test-curl-11348-4c1fae" Nov 03 16:59:56.116166 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:56.116149 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11348-c85649" Nov 03 16:59:56.134212 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:56.134189 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11348-c85649"] Nov 03 16:59:56.166040 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:56.166014 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf6sj\" (UniqueName: \"kubernetes.io/projected/568cdb76-88bb-49c2-a6ac-e453edd6eea5-kube-api-access-qf6sj\") pod \"test-curl-11348-c85649\" (UID: \"568cdb76-88bb-49c2-a6ac-e453edd6eea5\") " pod="opendatahub/test-curl-11348-c85649" Nov 03 16:59:56.267749 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:56.267691 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf6sj\" (UniqueName: \"kubernetes.io/projected/568cdb76-88bb-49c2-a6ac-e453edd6eea5-kube-api-access-qf6sj\") pod \"test-curl-11348-c85649\" (UID: \"568cdb76-88bb-49c2-a6ac-e453edd6eea5\") " pod="opendatahub/test-curl-11348-c85649" Nov 03 16:59:56.292829 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:56.292796 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf6sj\" (UniqueName: \"kubernetes.io/projected/568cdb76-88bb-49c2-a6ac-e453edd6eea5-kube-api-access-qf6sj\") pod \"test-curl-11348-c85649\" (UID: \"568cdb76-88bb-49c2-a6ac-e453edd6eea5\") " pod="opendatahub/test-curl-11348-c85649" Nov 03 16:59:56.429238 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:56.429202 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11348-c85649" Nov 03 16:59:56.766815 ip-10-0-143-64 kubenswrapper[4947]: W1103 16:59:56.766781 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod568cdb76_88bb_49c2_a6ac_e453edd6eea5.slice/crio-b3d30cd380bf2815daf89e9fc0f65fe33d635020e278d35cc536f8b5b8386e77 WatchSource:0}: Error finding container b3d30cd380bf2815daf89e9fc0f65fe33d635020e278d35cc536f8b5b8386e77: Status 404 returned error can't find the container with id b3d30cd380bf2815daf89e9fc0f65fe33d635020e278d35cc536f8b5b8386e77 Nov 03 16:59:56.767869 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:56.767843 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11348-c85649"] Nov 03 16:59:56.805951 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:56.805923 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11348-c85649" event={"ID":"568cdb76-88bb-49c2-a6ac-e453edd6eea5","Type":"ContainerStarted","Data":"b3d30cd380bf2815daf89e9fc0f65fe33d635020e278d35cc536f8b5b8386e77"} Nov 03 16:59:56.806063 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:56.805995 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11348-4c1fae" podUID="eaf834a9-d570-42dc-b01f-3a7565bf187d" containerName="test-curl-11348-4c1fae" containerID="cri-o://758f17e83dc5d2406bd5fed8ea0682faad7563cc5a703145136c01854590d05c" gracePeriod=2 Nov 03 16:59:57.073992 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:57.073968 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11348-dc6313" Nov 03 16:59:57.182137 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:57.182105 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d75eed1-8b6d-4e3c-af3c-57516addc1f1-kube-api-access-654st\" (UniqueName: \"kubernetes.io/projected/4d75eed1-8b6d-4e3c-af3c-57516addc1f1-kube-api-access-654st\") pod \"4d75eed1-8b6d-4e3c-af3c-57516addc1f1\" (UID: \"4d75eed1-8b6d-4e3c-af3c-57516addc1f1\") " Nov 03 16:59:57.184332 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:57.184308 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d75eed1-8b6d-4e3c-af3c-57516addc1f1-kube-api-access-654st" pod "4d75eed1-8b6d-4e3c-af3c-57516addc1f1" (UID: "4d75eed1-8b6d-4e3c-af3c-57516addc1f1"). InnerVolumeSpecName "kube-api-access-654st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:57.284427 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:57.284364 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-654st\" (UniqueName: \"kubernetes.io/projected/4d75eed1-8b6d-4e3c-af3c-57516addc1f1-kube-api-access-654st\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:57.813765 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:57.813723 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11348-c85649" event={"ID":"568cdb76-88bb-49c2-a6ac-e453edd6eea5","Type":"ContainerStarted","Data":"ae0a67f9313ba941569c7dea3009698bafb029ef3d13c0cf22e2b92ffd760550"} Nov 03 16:59:57.815271 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:57.815245 4947 generic.go:356] "Generic (PLEG): container finished" podID="4d75eed1-8b6d-4e3c-af3c-57516addc1f1" containerID="717c3a91f2df2abc09d643cd0c97a56898c98cf8c4434b6af1068f99efbfdcab" exitCode=137 Nov 03 16:59:57.815395 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:57.815280 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11348-dc6313" Nov 03 16:59:57.815395 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:57.815287 4947 scope.go:122] "RemoveContainer" containerID="717c3a91f2df2abc09d643cd0c97a56898c98cf8c4434b6af1068f99efbfdcab" Nov 03 16:59:57.826471 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:57.826453 4947 scope.go:122] "RemoveContainer" containerID="717c3a91f2df2abc09d643cd0c97a56898c98cf8c4434b6af1068f99efbfdcab" Nov 03 16:59:57.826778 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:59:57.826758 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"717c3a91f2df2abc09d643cd0c97a56898c98cf8c4434b6af1068f99efbfdcab\": container with ID starting with 717c3a91f2df2abc09d643cd0c97a56898c98cf8c4434b6af1068f99efbfdcab not found: ID does not exist" containerID="717c3a91f2df2abc09d643cd0c97a56898c98cf8c4434b6af1068f99efbfdcab" Nov 03 16:59:57.826840 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:57.826786 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"717c3a91f2df2abc09d643cd0c97a56898c98cf8c4434b6af1068f99efbfdcab"} err="failed to get container status \"717c3a91f2df2abc09d643cd0c97a56898c98cf8c4434b6af1068f99efbfdcab\": rpc error: code = NotFound desc = could not find container \"717c3a91f2df2abc09d643cd0c97a56898c98cf8c4434b6af1068f99efbfdcab\": container with ID starting with 717c3a91f2df2abc09d643cd0c97a56898c98cf8c4434b6af1068f99efbfdcab not found: ID does not exist" Nov 03 16:59:57.861618 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:57.861573 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11348-c85649" podStartSLOduration=1.8615582769999999 podStartE2EDuration="1.861558277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 16:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 16:59:57.858573338 +0000 UTC m=+2741.447573305" watchObservedRunningTime="2026-11-03 16:59:57.861558277 +0000 UTC m=+2741.450558246" Nov 03 16:59:57.882409 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:59:57.882378 4947 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11348-dc6313\" is forbidden: User \"system:node:ip-10-0-143-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-64.ec2.internal' and this object" podUID="4d75eed1-8b6d-4e3c-af3c-57516addc1f1" pod="opendatahub/test-curl-11348-dc6313" Nov 03 16:59:58.567079 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:58.567042 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d75eed1-8b6d-4e3c-af3c-57516addc1f1" path="/var/lib/kubelet/pods/4d75eed1-8b6d-4e3c-af3c-57516addc1f1/volumes" Nov 03 16:59:58.759345 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:58.759315 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11348-c85649"] Nov 03 16:59:58.763477 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:58.763449 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11348-c85649"] Nov 03 16:59:59.071542 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:59.071485 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11348-4c1fae" Nov 03 16:59:59.153511 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:59.153481 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf834a9-d570-42dc-b01f-3a7565bf187d-kube-api-access-jp9mm\" (UniqueName: \"kubernetes.io/projected/eaf834a9-d570-42dc-b01f-3a7565bf187d-kube-api-access-jp9mm\") pod \"eaf834a9-d570-42dc-b01f-3a7565bf187d\" (UID: \"eaf834a9-d570-42dc-b01f-3a7565bf187d\") " Nov 03 16:59:59.155793 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:59.155767 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf834a9-d570-42dc-b01f-3a7565bf187d-kube-api-access-jp9mm" pod "eaf834a9-d570-42dc-b01f-3a7565bf187d" (UID: "eaf834a9-d570-42dc-b01f-3a7565bf187d"). InnerVolumeSpecName "kube-api-access-jp9mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 16:59:59.255894 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:59.255865 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp9mm\" (UniqueName: \"kubernetes.io/projected/eaf834a9-d570-42dc-b01f-3a7565bf187d-kube-api-access-jp9mm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 16:59:59.882214 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:59.882171 4947 generic.go:356] "Generic (PLEG): container finished" podID="eaf834a9-d570-42dc-b01f-3a7565bf187d" containerID="758f17e83dc5d2406bd5fed8ea0682faad7563cc5a703145136c01854590d05c" exitCode=137 Nov 03 16:59:59.882675 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:59.882242 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11348-4c1fae" Nov 03 16:59:59.882675 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:59.882475 4947 scope.go:122] "RemoveContainer" containerID="758f17e83dc5d2406bd5fed8ea0682faad7563cc5a703145136c01854590d05c" Nov 03 16:59:59.882675 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:59.882553 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11348-c85649" podUID="568cdb76-88bb-49c2-a6ac-e453edd6eea5" containerName="test-curl-11348-c85649" containerID="cri-o://ae0a67f9313ba941569c7dea3009698bafb029ef3d13c0cf22e2b92ffd760550" gracePeriod=2 Nov 03 16:59:59.893246 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:59.893222 4947 scope.go:122] "RemoveContainer" containerID="758f17e83dc5d2406bd5fed8ea0682faad7563cc5a703145136c01854590d05c" Nov 03 16:59:59.893527 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:59:59.893507 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"758f17e83dc5d2406bd5fed8ea0682faad7563cc5a703145136c01854590d05c\": container with ID starting with 758f17e83dc5d2406bd5fed8ea0682faad7563cc5a703145136c01854590d05c not found: ID does not exist" containerID="758f17e83dc5d2406bd5fed8ea0682faad7563cc5a703145136c01854590d05c" Nov 03 16:59:59.893577 ip-10-0-143-64 kubenswrapper[4947]: I1103 16:59:59.893536 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"758f17e83dc5d2406bd5fed8ea0682faad7563cc5a703145136c01854590d05c"} err="failed to get container status \"758f17e83dc5d2406bd5fed8ea0682faad7563cc5a703145136c01854590d05c\": rpc error: code = NotFound desc = could not find container \"758f17e83dc5d2406bd5fed8ea0682faad7563cc5a703145136c01854590d05c\": container with ID starting with 758f17e83dc5d2406bd5fed8ea0682faad7563cc5a703145136c01854590d05c not found: ID does not exist" Nov 03 16:59:59.899156 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:59:59.899135 4947 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11348-c85649\" is forbidden: User \"system:node:ip-10-0-143-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-64.ec2.internal' and this object" podUID="568cdb76-88bb-49c2-a6ac-e453edd6eea5" pod="opendatahub/test-curl-11348-c85649" Nov 03 16:59:59.914233 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:59:59.914196 4947 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11348-c85649\" is forbidden: User \"system:node:ip-10-0-143-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-64.ec2.internal' and this object" podUID="568cdb76-88bb-49c2-a6ac-e453edd6eea5" pod="opendatahub/test-curl-11348-c85649" Nov 03 16:59:59.931684 ip-10-0-143-64 kubenswrapper[4947]: E1103 16:59:59.931656 4947 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11348-4c1fae\" is forbidden: User \"system:node:ip-10-0-143-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-64.ec2.internal' and this object" podUID="eaf834a9-d570-42dc-b01f-3a7565bf187d" pod="opendatahub/test-curl-11348-4c1fae" Nov 03 17:00:00.130373 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:00:00.130336 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:00:00.371859 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.371830 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs"] Nov 03 17:00:00.372562 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.372545 4947 cpu_manager.go:414] "containerMap: removing container" podUID="568cdb76-88bb-49c2-a6ac-e453edd6eea5" containerName="test-curl-11348-c85649" Nov 03 17:00:00.372662 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.372562 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="568cdb76-88bb-49c2-a6ac-e453edd6eea5" containerName="test-curl-11348-c85649" Nov 03 17:00:00.372762 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.372690 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="568cdb76-88bb-49c2-a6ac-e453edd6eea5" containerName="test-curl-11348-c85649" Nov 03 17:00:00.378397 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.378374 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" Nov 03 17:00:00.393479 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:00:00.393449 4947 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11348-c85649\" is forbidden: User \"system:node:ip-10-0-143-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-64.ec2.internal' and this object" podUID="568cdb76-88bb-49c2-a6ac-e453edd6eea5" pod="opendatahub/test-curl-11348-c85649" Nov 03 17:00:00.411046 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:00:00.411010 4947 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11348-4c1fae\" is forbidden: User \"system:node:ip-10-0-143-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-64.ec2.internal' and this object" podUID="eaf834a9-d570-42dc-b01f-3a7565bf187d" pod="opendatahub/test-curl-11348-4c1fae" Nov 03 17:00:00.412536 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.412268 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs"] Nov 03 17:00:00.416469 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.416449 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg"] Nov 03 17:00:00.421095 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.421076 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" Nov 03 17:00:00.453091 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.453062 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg"] Nov 03 17:00:00.465599 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.465563 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:00:00.474658 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.474629 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtfwt\" (UniqueName: \"kubernetes.io/projected/9fb5522f-9240-4c7b-bc72-4fa24c5e1b99-kube-api-access-xtfwt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg\" (UID: \"9fb5522f-9240-4c7b-bc72-4fa24c5e1b99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" Nov 03 17:00:00.474853 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.474747 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs\" (UID: \"ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" Nov 03 17:00:00.474853 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.474835 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fb5522f-9240-4c7b-bc72-4fa24c5e1b99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg\" (UID: \"9fb5522f-9240-4c7b-bc72-4fa24c5e1b99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" Nov 03 17:00:00.474979 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.474900 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnk9p\" (UniqueName: \"kubernetes.io/projected/ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47-kube-api-access-fnk9p\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs\" (UID: \"ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" Nov 03 17:00:00.481975 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:00:00.481943 4947 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11348-c85649\" is forbidden: User \"system:node:ip-10-0-143-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-64.ec2.internal' and this object" podUID="568cdb76-88bb-49c2-a6ac-e453edd6eea5" pod="opendatahub/test-curl-11348-c85649" Nov 03 17:00:00.497365 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:00:00.497337 4947 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11348-4c1fae\" is forbidden: User \"system:node:ip-10-0-143-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-64.ec2.internal' and this object" podUID="eaf834a9-d570-42dc-b01f-3a7565bf187d" pod="opendatahub/test-curl-11348-4c1fae" Nov 03 17:00:00.567121 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.567084 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf834a9-d570-42dc-b01f-3a7565bf187d" path="/var/lib/kubelet/pods/eaf834a9-d570-42dc-b01f-3a7565bf187d/volumes" Nov 03 17:00:00.575951 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.575925 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtfwt\" (UniqueName: \"kubernetes.io/projected/9fb5522f-9240-4c7b-bc72-4fa24c5e1b99-kube-api-access-xtfwt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg\" (UID: \"9fb5522f-9240-4c7b-bc72-4fa24c5e1b99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" Nov 03 17:00:00.576089 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.575981 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs\" (UID: \"ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" Nov 03 17:00:00.576089 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.576031 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fb5522f-9240-4c7b-bc72-4fa24c5e1b99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg\" (UID: \"9fb5522f-9240-4c7b-bc72-4fa24c5e1b99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" Nov 03 17:00:00.576089 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.576068 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnk9p\" (UniqueName: \"kubernetes.io/projected/ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47-kube-api-access-fnk9p\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs\" (UID: \"ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" Nov 03 17:00:00.576784 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.576760 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs\" (UID: \"ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" Nov 03 17:00:00.576878 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.576760 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fb5522f-9240-4c7b-bc72-4fa24c5e1b99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg\" (UID: \"9fb5522f-9240-4c7b-bc72-4fa24c5e1b99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" Nov 03 17:00:00.604203 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.604170 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnk9p\" (UniqueName: \"kubernetes.io/projected/ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47-kube-api-access-fnk9p\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs\" (UID: \"ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" Nov 03 17:00:00.604344 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.604170 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtfwt\" (UniqueName: \"kubernetes.io/projected/9fb5522f-9240-4c7b-bc72-4fa24c5e1b99-kube-api-access-xtfwt\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg\" (UID: \"9fb5522f-9240-4c7b-bc72-4fa24c5e1b99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" Nov 03 17:00:00.695483 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.695413 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" Nov 03 17:00:00.740894 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:00.740865 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" Nov 03 17:00:01.049413 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:00:01.049380 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9ff523_dcb8_42d3_b7b4_dc2734a6ca47.slice/crio-c5d54c533f5ae4693c76b7cd2fa18ad086916e1c1b66edb83d057229229f9953 WatchSource:0}: Error finding container c5d54c533f5ae4693c76b7cd2fa18ad086916e1c1b66edb83d057229229f9953: Status 404 returned error can't find the container with id c5d54c533f5ae4693c76b7cd2fa18ad086916e1c1b66edb83d057229229f9953 Nov 03 17:00:01.054105 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:01.054081 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs"] Nov 03 17:00:01.102398 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:00:01.102373 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb5522f_9240_4c7b_bc72_4fa24c5e1b99.slice/crio-e12eb2b63baece2d46d64ebe326c69d048db1747ec0f6a011e6eca9d36dec38e WatchSource:0}: Error finding container e12eb2b63baece2d46d64ebe326c69d048db1747ec0f6a011e6eca9d36dec38e: Status 404 returned error can't find the container with id e12eb2b63baece2d46d64ebe326c69d048db1747ec0f6a011e6eca9d36dec38e Nov 03 17:00:01.103190 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:01.103165 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg"] Nov 03 17:00:01.904010 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:01.903981 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" event={"ID":"ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47","Type":"ContainerStarted","Data":"c5d54c533f5ae4693c76b7cd2fa18ad086916e1c1b66edb83d057229229f9953"} Nov 03 17:00:01.905294 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:01.905266 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" event={"ID":"9fb5522f-9240-4c7b-bc72-4fa24c5e1b99","Type":"ContainerStarted","Data":"e12eb2b63baece2d46d64ebe326c69d048db1747ec0f6a011e6eca9d36dec38e"} Nov 03 17:00:02.080268 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:02.080243 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11348-c85649" Nov 03 17:00:02.109911 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:02.109887 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/568cdb76-88bb-49c2-a6ac-e453edd6eea5-kube-api-access-qf6sj\" (UniqueName: \"kubernetes.io/projected/568cdb76-88bb-49c2-a6ac-e453edd6eea5-kube-api-access-qf6sj\") pod \"568cdb76-88bb-49c2-a6ac-e453edd6eea5\" (UID: \"568cdb76-88bb-49c2-a6ac-e453edd6eea5\") " Nov 03 17:00:02.112053 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:02.112026 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/568cdb76-88bb-49c2-a6ac-e453edd6eea5-kube-api-access-qf6sj" pod "568cdb76-88bb-49c2-a6ac-e453edd6eea5" (UID: "568cdb76-88bb-49c2-a6ac-e453edd6eea5"). InnerVolumeSpecName "kube-api-access-qf6sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:02.211823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:02.211760 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf6sj\" (UniqueName: \"kubernetes.io/projected/568cdb76-88bb-49c2-a6ac-e453edd6eea5-kube-api-access-qf6sj\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:02.569938 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:02.569904 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="568cdb76-88bb-49c2-a6ac-e453edd6eea5" path="/var/lib/kubelet/pods/568cdb76-88bb-49c2-a6ac-e453edd6eea5/volumes" Nov 03 17:00:02.787720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:02.787358 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:00:02.923175 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:02.922996 4947 generic.go:356] "Generic (PLEG): container finished" podID="568cdb76-88bb-49c2-a6ac-e453edd6eea5" containerID="ae0a67f9313ba941569c7dea3009698bafb029ef3d13c0cf22e2b92ffd760550" exitCode=137 Nov 03 17:00:02.923175 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:02.923115 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11348-c85649" Nov 03 17:00:02.923633 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:02.923616 4947 scope.go:122] "RemoveContainer" containerID="ae0a67f9313ba941569c7dea3009698bafb029ef3d13c0cf22e2b92ffd760550" Nov 03 17:00:02.938256 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:02.938230 4947 scope.go:122] "RemoveContainer" containerID="ae0a67f9313ba941569c7dea3009698bafb029ef3d13c0cf22e2b92ffd760550" Nov 03 17:00:02.938587 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:00:02.938562 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae0a67f9313ba941569c7dea3009698bafb029ef3d13c0cf22e2b92ffd760550\": container with ID starting with ae0a67f9313ba941569c7dea3009698bafb029ef3d13c0cf22e2b92ffd760550 not found: ID does not exist" containerID="ae0a67f9313ba941569c7dea3009698bafb029ef3d13c0cf22e2b92ffd760550" Nov 03 17:00:02.938669 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:02.938598 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae0a67f9313ba941569c7dea3009698bafb029ef3d13c0cf22e2b92ffd760550"} err="failed to get container status \"ae0a67f9313ba941569c7dea3009698bafb029ef3d13c0cf22e2b92ffd760550\": rpc error: code = NotFound desc = could not find container \"ae0a67f9313ba941569c7dea3009698bafb029ef3d13c0cf22e2b92ffd760550\": container with ID starting with ae0a67f9313ba941569c7dea3009698bafb029ef3d13c0cf22e2b92ffd760550 not found: ID does not exist" Nov 03 17:00:05.953159 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:05.952990 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" event={"ID":"ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47","Type":"ContainerStarted","Data":"fd615212ebb24066ec5cbb1f57e617e75e7c0c2cdae63ffd3a4923eabf28802f"} Nov 03 17:00:06.014756 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:06.014685 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" podStartSLOduration=1.80711515 podStartE2EDuration="6.014671272s" totalImagesPullingTime="4.207556122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:06.014011836 +0000 UTC m=+2749.603011803" watchObservedRunningTime="2026-11-03 17:00:06.014671272 +0000 UTC m=+2749.603671238" Nov 03 17:00:06.180054 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:00:06.180016 4947 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9ff523_dcb8_42d3_b7b4_dc2734a6ca47.slice/crio-fd615212ebb24066ec5cbb1f57e617e75e7c0c2cdae63ffd3a4923eabf28802f.scope\": RecentStats: unable to find data in memory cache]" Nov 03 17:00:06.961826 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:06.961794 4947 generic.go:356] "Generic (PLEG): container finished" podID="ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47" containerID="fd615212ebb24066ec5cbb1f57e617e75e7c0c2cdae63ffd3a4923eabf28802f" exitCode=60 Nov 03 17:00:06.962305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:06.961883 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" event={"ID":"ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47","Type":"ContainerDied","Data":"fd615212ebb24066ec5cbb1f57e617e75e7c0c2cdae63ffd3a4923eabf28802f"} Nov 03 17:00:06.962305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:06.962226 4947 scope.go:122] "RemoveContainer" containerID="fd615212ebb24066ec5cbb1f57e617e75e7c0c2cdae63ffd3a4923eabf28802f" Nov 03 17:00:07.972721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:07.972667 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" event={"ID":"ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47","Type":"ContainerStarted","Data":"8d17a0b85c38a91dea94d232bd8cb98bca6b6c539a12a2c521173ece1a19138a"} Nov 03 17:00:08.981707 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:08.981663 4947 generic.go:356] "Generic (PLEG): container finished" podID="ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47" containerID="8d17a0b85c38a91dea94d232bd8cb98bca6b6c539a12a2c521173ece1a19138a" exitCode=60 Nov 03 17:00:08.982135 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:08.981730 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" event={"ID":"ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47","Type":"ContainerDied","Data":"8d17a0b85c38a91dea94d232bd8cb98bca6b6c539a12a2c521173ece1a19138a"} Nov 03 17:00:08.982135 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:08.981787 4947 scope.go:122] "RemoveContainer" containerID="fd615212ebb24066ec5cbb1f57e617e75e7c0c2cdae63ffd3a4923eabf28802f" Nov 03 17:00:08.982135 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:08.982103 4947 scope.go:122] "RemoveContainer" containerID="8d17a0b85c38a91dea94d232bd8cb98bca6b6c539a12a2c521173ece1a19138a" Nov 03 17:00:08.982494 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:00:08.982470 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs_odh-ai-gateway-infra(ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" podUID="ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47" Nov 03 17:00:10.062373 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:10.061527 4947 scope.go:122] "RemoveContainer" containerID="8d17a0b85c38a91dea94d232bd8cb98bca6b6c539a12a2c521173ece1a19138a" Nov 03 17:00:10.062373 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:00:10.061987 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs_odh-ai-gateway-infra(ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" podUID="ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47" Nov 03 17:00:10.075406 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:10.075329 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" event={"ID":"9fb5522f-9240-4c7b-bc72-4fa24c5e1b99","Type":"ContainerStarted","Data":"70537a3d3566a832a4b6e9e2a2e397dc8c716275b5d41a05086a4ba44f65ddfa"} Nov 03 17:00:10.173189 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:10.173132 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" podStartSLOduration=2.149783601 podStartE2EDuration="10.173117963s" totalImagesPullingTime="8.023334362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:10.170692 +0000 UTC m=+2753.759691968" watchObservedRunningTime="2026-11-03 17:00:10.173117963 +0000 UTC m=+2753.762117972" Nov 03 17:00:11.085101 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:11.085065 4947 generic.go:356] "Generic (PLEG): container finished" podID="9fb5522f-9240-4c7b-bc72-4fa24c5e1b99" containerID="70537a3d3566a832a4b6e9e2a2e397dc8c716275b5d41a05086a4ba44f65ddfa" exitCode=60 Nov 03 17:00:11.085549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:11.085144 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" event={"ID":"9fb5522f-9240-4c7b-bc72-4fa24c5e1b99","Type":"ContainerDied","Data":"70537a3d3566a832a4b6e9e2a2e397dc8c716275b5d41a05086a4ba44f65ddfa"} Nov 03 17:00:11.085549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:11.085445 4947 scope.go:122] "RemoveContainer" containerID="70537a3d3566a832a4b6e9e2a2e397dc8c716275b5d41a05086a4ba44f65ddfa" Nov 03 17:00:12.094482 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:12.094446 4947 generic.go:356] "Generic (PLEG): container finished" podID="9fb5522f-9240-4c7b-bc72-4fa24c5e1b99" containerID="7c0b95ece65ef081f0ab336a8921c3120e0c88163b6958827bc4345b12475d18" exitCode=60 Nov 03 17:00:12.094908 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:12.094541 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" event={"ID":"9fb5522f-9240-4c7b-bc72-4fa24c5e1b99","Type":"ContainerDied","Data":"7c0b95ece65ef081f0ab336a8921c3120e0c88163b6958827bc4345b12475d18"} Nov 03 17:00:12.094908 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:12.094602 4947 scope.go:122] "RemoveContainer" containerID="70537a3d3566a832a4b6e9e2a2e397dc8c716275b5d41a05086a4ba44f65ddfa" Nov 03 17:00:12.094908 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:12.094881 4947 scope.go:122] "RemoveContainer" containerID="7c0b95ece65ef081f0ab336a8921c3120e0c88163b6958827bc4345b12475d18" Nov 03 17:00:12.095281 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:00:12.095249 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg_odh-ai-gateway-infra(9fb5522f-9240-4c7b-bc72-4fa24c5e1b99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" podUID="9fb5522f-9240-4c7b-bc72-4fa24c5e1b99" Nov 03 17:00:13.107555 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:13.107523 4947 scope.go:122] "RemoveContainer" containerID="7c0b95ece65ef081f0ab336a8921c3120e0c88163b6958827bc4345b12475d18" Nov 03 17:00:13.108055 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:00:13.107980 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg_odh-ai-gateway-infra(9fb5522f-9240-4c7b-bc72-4fa24c5e1b99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" podUID="9fb5522f-9240-4c7b-bc72-4fa24c5e1b99" Nov 03 17:00:19.558683 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:19.558652 4947 scope.go:122] "RemoveContainer" containerID="8d17a0b85c38a91dea94d232bd8cb98bca6b6c539a12a2c521173ece1a19138a" Nov 03 17:00:20.158936 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:20.158856 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" event={"ID":"ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47","Type":"ContainerStarted","Data":"fb5388f3ec726d4677a784d2008411f743be060a357f5bf24b6b85aae8b7135c"} Nov 03 17:00:21.167014 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:21.166977 4947 generic.go:356] "Generic (PLEG): container finished" podID="ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47" containerID="fb5388f3ec726d4677a784d2008411f743be060a357f5bf24b6b85aae8b7135c" exitCode=60 Nov 03 17:00:21.167499 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:21.167044 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" event={"ID":"ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47","Type":"ContainerDied","Data":"fb5388f3ec726d4677a784d2008411f743be060a357f5bf24b6b85aae8b7135c"} Nov 03 17:00:21.167499 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:21.167103 4947 scope.go:122] "RemoveContainer" containerID="8d17a0b85c38a91dea94d232bd8cb98bca6b6c539a12a2c521173ece1a19138a" Nov 03 17:00:21.167606 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:21.167567 4947 scope.go:122] "RemoveContainer" containerID="fb5388f3ec726d4677a784d2008411f743be060a357f5bf24b6b85aae8b7135c" Nov 03 17:00:21.167970 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:00:21.167945 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs_odh-ai-gateway-infra(ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" podUID="ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47" Nov 03 17:00:21.219823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:21.219792 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs"] Nov 03 17:00:22.501228 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:22.501190 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" Nov 03 17:00:22.565568 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:22.565532 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47-maas-api-service-ca\") pod \"ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47\" (UID: \"ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47\") " Nov 03 17:00:22.565888 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:22.565867 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47-kube-api-access-fnk9p\" (UniqueName: \"kubernetes.io/projected/ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47-kube-api-access-fnk9p\") pod \"ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47\" (UID: \"ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47\") " Nov 03 17:00:22.568450 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:22.568425 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47-maas-api-service-ca" pod "ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47" (UID: "ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:22.569216 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:22.569194 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47-kube-api-access-fnk9p" pod "ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47" (UID: "ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47"). InnerVolumeSpecName "kube-api-access-fnk9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:22.667496 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:22.667457 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47-maas-api-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:22.667496 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:22.667484 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnk9p\" (UniqueName: \"kubernetes.io/projected/ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47-kube-api-access-fnk9p\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:23.222845 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:23.222812 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" Nov 03 17:00:23.223048 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:23.222845 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs" event={"ID":"ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47","Type":"ContainerDied","Data":"c5d54c533f5ae4693c76b7cd2fa18ad086916e1c1b66edb83d057229229f9953"} Nov 03 17:00:23.223048 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:23.222942 4947 scope.go:122] "RemoveContainer" containerID="fb5388f3ec726d4677a784d2008411f743be060a357f5bf24b6b85aae8b7135c" Nov 03 17:00:23.288341 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:23.288306 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs"] Nov 03 17:00:23.299521 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:23.299496 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895420-qlhjs"] Nov 03 17:00:23.807851 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:23.807807 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k"] Nov 03 17:00:23.818734 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:23.809578 4947 cpu_manager.go:414] "containerMap: removing container" podUID="ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47" containerName="cleanup" Nov 03 17:00:23.818978 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:23.818949 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47" containerName="cleanup" Nov 03 17:00:23.819237 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:23.819220 4947 cpu_manager.go:414] "containerMap: removing container" podUID="ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47" containerName="cleanup" Nov 03 17:00:23.819375 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:23.819359 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47" containerName="cleanup" Nov 03 17:00:23.819485 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:23.819471 4947 cpu_manager.go:414] "containerMap: removing container" podUID="ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47" containerName="cleanup" Nov 03 17:00:23.819566 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:23.819555 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47" containerName="cleanup" Nov 03 17:00:23.819908 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:23.819886 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47" containerName="cleanup" Nov 03 17:00:23.820045 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:23.820033 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47" containerName="cleanup" Nov 03 17:00:23.871234 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:23.871202 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" Nov 03 17:00:23.926797 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:23.918557 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-da409b-dockercfg-v7g4m\"" Nov 03 17:00:23.965723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:23.927883 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-da409b\"" Nov 03 17:00:24.012137 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:24.000495 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k"] Nov 03 17:00:24.068983 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:24.068888 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx8h5\" (UniqueName: \"kubernetes.io/projected/8d41dd69-ceef-4cd2-90a3-099d4562d32f-kube-api-access-vx8h5\") pod \"payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k\" (UID: \"8d41dd69-ceef-4cd2-90a3-099d4562d32f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" Nov 03 17:00:24.069171 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:24.068984 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d41dd69-ceef-4cd2-90a3-099d4562d32f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k\" (UID: \"8d41dd69-ceef-4cd2-90a3-099d4562d32f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" Nov 03 17:00:24.170954 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:24.170914 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx8h5\" (UniqueName: \"kubernetes.io/projected/8d41dd69-ceef-4cd2-90a3-099d4562d32f-kube-api-access-vx8h5\") pod \"payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k\" (UID: \"8d41dd69-ceef-4cd2-90a3-099d4562d32f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" Nov 03 17:00:24.171126 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:24.170978 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d41dd69-ceef-4cd2-90a3-099d4562d32f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k\" (UID: \"8d41dd69-ceef-4cd2-90a3-099d4562d32f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" Nov 03 17:00:24.171638 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:24.171619 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d41dd69-ceef-4cd2-90a3-099d4562d32f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k\" (UID: \"8d41dd69-ceef-4cd2-90a3-099d4562d32f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" Nov 03 17:00:24.196055 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:24.196023 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx8h5\" (UniqueName: \"kubernetes.io/projected/8d41dd69-ceef-4cd2-90a3-099d4562d32f-kube-api-access-vx8h5\") pod \"payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k\" (UID: \"8d41dd69-ceef-4cd2-90a3-099d4562d32f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" Nov 03 17:00:24.323614 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:24.323535 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" Nov 03 17:00:24.567126 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:24.567096 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47" path="/var/lib/kubelet/pods/ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47/volumes" Nov 03 17:00:24.635556 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:00:24.635526 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d41dd69_ceef_4cd2_90a3_099d4562d32f.slice/crio-6c8aed15e13716b525e7b59796f30cdbebd09e1931f9c1cf28f11440986d5e9a WatchSource:0}: Error finding container 6c8aed15e13716b525e7b59796f30cdbebd09e1931f9c1cf28f11440986d5e9a: Status 404 returned error can't find the container with id 6c8aed15e13716b525e7b59796f30cdbebd09e1931f9c1cf28f11440986d5e9a Nov 03 17:00:24.636059 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:24.636034 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k"] Nov 03 17:00:25.240665 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:25.240620 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" event={"ID":"8d41dd69-ceef-4cd2-90a3-099d4562d32f","Type":"ContainerStarted","Data":"b03c2f46f85885b4fe09d2dbb791e8196cd016a7aeb8fd02d86d0c8eb3e79c5b"} Nov 03 17:00:25.241183 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:25.240687 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" event={"ID":"8d41dd69-ceef-4cd2-90a3-099d4562d32f","Type":"ContainerStarted","Data":"6c8aed15e13716b525e7b59796f30cdbebd09e1931f9c1cf28f11440986d5e9a"} Nov 03 17:00:25.241183 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:25.240740 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" Nov 03 17:00:25.285661 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:25.285609 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" podStartSLOduration=2.285596772 podStartE2EDuration="2.285596772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:00:25.284202943 +0000 UTC m=+2768.873202911" watchObservedRunningTime="2026-11-03 17:00:25.285596772 +0000 UTC m=+2768.874596739" Nov 03 17:00:26.562045 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:26.562014 4947 scope.go:122] "RemoveContainer" containerID="7c0b95ece65ef081f0ab336a8921c3120e0c88163b6958827bc4345b12475d18" Nov 03 17:00:27.256159 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:27.256127 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" event={"ID":"9fb5522f-9240-4c7b-bc72-4fa24c5e1b99","Type":"ContainerStarted","Data":"1bda1f6d777744d080dcd663566bb15fbe4c230adb7b54263ed66db14668c3e4"} Nov 03 17:00:28.264084 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:28.264050 4947 generic.go:356] "Generic (PLEG): container finished" podID="9fb5522f-9240-4c7b-bc72-4fa24c5e1b99" containerID="1bda1f6d777744d080dcd663566bb15fbe4c230adb7b54263ed66db14668c3e4" exitCode=60 Nov 03 17:00:28.264514 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:28.264130 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" event={"ID":"9fb5522f-9240-4c7b-bc72-4fa24c5e1b99","Type":"ContainerDied","Data":"1bda1f6d777744d080dcd663566bb15fbe4c230adb7b54263ed66db14668c3e4"} Nov 03 17:00:28.264514 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:28.264187 4947 scope.go:122] "RemoveContainer" containerID="7c0b95ece65ef081f0ab336a8921c3120e0c88163b6958827bc4345b12475d18" Nov 03 17:00:28.264655 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:28.264565 4947 scope.go:122] "RemoveContainer" containerID="1bda1f6d777744d080dcd663566bb15fbe4c230adb7b54263ed66db14668c3e4" Nov 03 17:00:28.265014 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:00:28.264971 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg_odh-ai-gateway-infra(9fb5522f-9240-4c7b-bc72-4fa24c5e1b99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" podUID="9fb5522f-9240-4c7b-bc72-4fa24c5e1b99" Nov 03 17:00:28.326683 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:28.326655 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg"] Nov 03 17:00:29.501213 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:29.500764 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" Nov 03 17:00:29.703370 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:29.703280 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb5522f-9240-4c7b-bc72-4fa24c5e1b99-kube-api-access-xtfwt\" (UniqueName: \"kubernetes.io/projected/9fb5522f-9240-4c7b-bc72-4fa24c5e1b99-kube-api-access-xtfwt\") pod \"9fb5522f-9240-4c7b-bc72-4fa24c5e1b99\" (UID: \"9fb5522f-9240-4c7b-bc72-4fa24c5e1b99\") " Nov 03 17:00:29.703542 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:29.703484 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fb5522f-9240-4c7b-bc72-4fa24c5e1b99-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fb5522f-9240-4c7b-bc72-4fa24c5e1b99-maas-api-service-ca\") pod \"9fb5522f-9240-4c7b-bc72-4fa24c5e1b99\" (UID: \"9fb5522f-9240-4c7b-bc72-4fa24c5e1b99\") " Nov 03 17:00:29.703860 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:29.703835 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb5522f-9240-4c7b-bc72-4fa24c5e1b99-maas-api-service-ca" pod "9fb5522f-9240-4c7b-bc72-4fa24c5e1b99" (UID: "9fb5522f-9240-4c7b-bc72-4fa24c5e1b99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:29.704204 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:29.704184 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fb5522f-9240-4c7b-bc72-4fa24c5e1b99-maas-api-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:29.705316 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:29.705296 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb5522f-9240-4c7b-bc72-4fa24c5e1b99-kube-api-access-xtfwt" pod "9fb5522f-9240-4c7b-bc72-4fa24c5e1b99" (UID: "9fb5522f-9240-4c7b-bc72-4fa24c5e1b99"). InnerVolumeSpecName "kube-api-access-xtfwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:29.805453 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:29.805420 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtfwt\" (UniqueName: \"kubernetes.io/projected/9fb5522f-9240-4c7b-bc72-4fa24c5e1b99-kube-api-access-xtfwt\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:30.133990 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:00:30.133954 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:00:30.282226 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:30.282198 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" Nov 03 17:00:30.282437 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:30.282238 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg" event={"ID":"9fb5522f-9240-4c7b-bc72-4fa24c5e1b99","Type":"ContainerDied","Data":"e12eb2b63baece2d46d64ebe326c69d048db1747ec0f6a011e6eca9d36dec38e"} Nov 03 17:00:30.282437 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:30.282305 4947 scope.go:122] "RemoveContainer" containerID="1bda1f6d777744d080dcd663566bb15fbe4c230adb7b54263ed66db14668c3e4" Nov 03 17:00:30.348087 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:30.348053 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg"] Nov 03 17:00:30.356737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:30.356688 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895420-hb5vg"] Nov 03 17:00:30.580071 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:30.579928 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb5522f-9240-4c7b-bc72-4fa24c5e1b99" path="/var/lib/kubelet/pods/9fb5522f-9240-4c7b-bc72-4fa24c5e1b99/volumes" Nov 03 17:00:30.808540 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:30.808507 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj"] Nov 03 17:00:36.026124 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:36.026078 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p"] Nov 03 17:00:36.026578 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:36.026348 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" podUID="b720f231-ec3b-4e94-bbf5-8920f8976a9e" containerName="istio-proxy" containerID="cri-o://b94f168503075af39a394b9607c2fe9e4f9062f20ae29b4075ad5a9bb230f286" gracePeriod=30 Nov 03 17:00:36.250779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:36.250751 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" Nov 03 17:00:40.283683 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:40.283600 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc"] Nov 03 17:00:40.284280 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:40.283954 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" podUID="1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" containerName="istio-proxy" containerID="cri-o://52aff2feb0520554e0777b6ae637b85f0b2abea2e89aa85b6ea4d5c2329a2384" gracePeriod=30 Nov 03 17:00:40.401938 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:40.401903 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:00:41.434964 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.434682 4947 generic.go:356] "Generic (PLEG): container finished" podID="b720f231-ec3b-4e94-bbf5-8920f8976a9e" containerID="b94f168503075af39a394b9607c2fe9e4f9062f20ae29b4075ad5a9bb230f286" exitCode=0 Nov 03 17:00:41.434964 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.434848 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" event={"ID":"b720f231-ec3b-4e94-bbf5-8920f8976a9e","Type":"ContainerDied","Data":"b94f168503075af39a394b9607c2fe9e4f9062f20ae29b4075ad5a9bb230f286"} Nov 03 17:00:41.468434 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.468401 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 17:00:41.560935 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.560901 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-workload-socket\") pod \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " Nov 03 17:00:41.561154 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.561008 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istiod-ca-cert\") pod \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " Nov 03 17:00:41.561154 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.561043 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b720f231-ec3b-4e94-bbf5-8920f8976a9e-kube-api-access-fwktw\" (UniqueName: \"kubernetes.io/projected/b720f231-ec3b-4e94-bbf5-8920f8976a9e-kube-api-access-fwktw\") pod \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " Nov 03 17:00:41.561154 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.561094 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-workload-certs\") pod \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " Nov 03 17:00:41.561154 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.561124 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-envoy\") pod \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " Nov 03 17:00:41.561369 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.561165 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-data\") pod \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " Nov 03 17:00:41.561369 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.561181 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-workload-socket" pod "b720f231-ec3b-4e94-bbf5-8920f8976a9e" (UID: "b720f231-ec3b-4e94-bbf5-8920f8976a9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:41.561369 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.561313 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-credential-socket\") pod \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " Nov 03 17:00:41.561514 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.561368 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-token\" (UniqueName: \"kubernetes.io/projected/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-token\") pod \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " Nov 03 17:00:41.561514 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.561411 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-workload-certs" pod "b720f231-ec3b-4e94-bbf5-8920f8976a9e" (UID: "b720f231-ec3b-4e94-bbf5-8920f8976a9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:41.561514 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.561457 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istiod-ca-cert" pod "b720f231-ec3b-4e94-bbf5-8920f8976a9e" (UID: "b720f231-ec3b-4e94-bbf5-8920f8976a9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:41.561662 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.561514 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-podinfo\") pod \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\" (UID: \"b720f231-ec3b-4e94-bbf5-8920f8976a9e\") " Nov 03 17:00:41.561662 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.561636 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-data" pod "b720f231-ec3b-4e94-bbf5-8920f8976a9e" (UID: "b720f231-ec3b-4e94-bbf5-8920f8976a9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:41.562202 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.562049 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-credential-socket" pod "b720f231-ec3b-4e94-bbf5-8920f8976a9e" (UID: "b720f231-ec3b-4e94-bbf5-8920f8976a9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:41.562202 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.562058 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:41.562202 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.562134 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:41.562202 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.562150 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:41.562202 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.562175 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:41.566643 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.566525 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-envoy" pod "b720f231-ec3b-4e94-bbf5-8920f8976a9e" (UID: "b720f231-ec3b-4e94-bbf5-8920f8976a9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:41.566643 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.566534 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-podinfo" pod "b720f231-ec3b-4e94-bbf5-8920f8976a9e" (UID: "b720f231-ec3b-4e94-bbf5-8920f8976a9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:00:41.566643 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.566560 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b720f231-ec3b-4e94-bbf5-8920f8976a9e-kube-api-access-fwktw" pod "b720f231-ec3b-4e94-bbf5-8920f8976a9e" (UID: "b720f231-ec3b-4e94-bbf5-8920f8976a9e"). InnerVolumeSpecName "kube-api-access-fwktw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:41.566643 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.566619 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-token" pod "b720f231-ec3b-4e94-bbf5-8920f8976a9e" (UID: "b720f231-ec3b-4e94-bbf5-8920f8976a9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:41.680745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.665748 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:41.680745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.665787 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:41.680745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.665801 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:41.680745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.665813 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwktw\" (UniqueName: \"kubernetes.io/projected/b720f231-ec3b-4e94-bbf5-8920f8976a9e-kube-api-access-fwktw\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:41.680745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:41.665826 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b720f231-ec3b-4e94-bbf5-8920f8976a9e-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:42.446093 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:42.446054 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" event={"ID":"b720f231-ec3b-4e94-bbf5-8920f8976a9e","Type":"ContainerDied","Data":"e87a26c86659f97049de5f3df8da3cf37986804329d1237caaa6c1ce32c44ee0"} Nov 03 17:00:42.446558 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:42.446131 4947 scope.go:122] "RemoveContainer" containerID="b94f168503075af39a394b9607c2fe9e4f9062f20ae29b4075ad5a9bb230f286" Nov 03 17:00:42.446558 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:42.446310 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p" Nov 03 17:00:42.522789 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:42.522748 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p"] Nov 03 17:00:42.544977 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:42.542348 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-v2p5p"] Nov 03 17:00:42.568518 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:42.568481 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b720f231-ec3b-4e94-bbf5-8920f8976a9e" path="/var/lib/kubelet/pods/b720f231-ec3b-4e94-bbf5-8920f8976a9e/volumes" Nov 03 17:00:43.955138 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:43.955097 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln"] Nov 03 17:00:43.955639 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:43.955346 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" podUID="036fe6c6-378c-42f3-87cc-91315033ea59" containerName="istio-proxy" containerID="cri-o://01426ee92fff2e74418abcd33b6851b12a1c7f0bdfa8ff416a86576178d0059b" gracePeriod=30 Nov 03 17:00:44.658779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:44.658746 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh"] Nov 03 17:00:45.478982 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.478780 4947 generic.go:356] "Generic (PLEG): container finished" podID="1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" containerID="52aff2feb0520554e0777b6ae637b85f0b2abea2e89aa85b6ea4d5c2329a2384" exitCode=0 Nov 03 17:00:45.478982 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.478925 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" event={"ID":"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e","Type":"ContainerDied","Data":"52aff2feb0520554e0777b6ae637b85f0b2abea2e89aa85b6ea4d5c2329a2384"} Nov 03 17:00:45.575274 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.575204 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 17:00:45.684386 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.684348 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-envoy\") pod \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " Nov 03 17:00:45.684609 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.684407 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-workload-certs\") pod \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " Nov 03 17:00:45.684609 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.684550 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-token\" (UniqueName: \"kubernetes.io/projected/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-token\") pod \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " Nov 03 17:00:45.684609 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.684597 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istiod-ca-cert\") pod \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " Nov 03 17:00:45.684832 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.684672 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-credential-socket\") pod \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " Nov 03 17:00:45.684832 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.684753 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-kube-api-access-2ng84\" (UniqueName: \"kubernetes.io/projected/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-kube-api-access-2ng84\") pod \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " Nov 03 17:00:45.684832 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.684786 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-workload-socket\") pod \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " Nov 03 17:00:45.684988 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.684838 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-data\") pod \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " Nov 03 17:00:45.684988 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.684974 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-podinfo\") pod \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\" (UID: \"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e\") " Nov 03 17:00:45.685140 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.685107 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-credential-socket" pod "1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" (UID: "1fb55a71-37cf-4f12-bb4e-6edef8a3db2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:45.685274 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.685251 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-workload-socket" pod "1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" (UID: "1fb55a71-37cf-4f12-bb4e-6edef8a3db2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:45.685489 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.685402 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-data" pod "1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" (UID: "1fb55a71-37cf-4f12-bb4e-6edef8a3db2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:45.685573 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.685484 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istiod-ca-cert" pod "1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" (UID: "1fb55a71-37cf-4f12-bb4e-6edef8a3db2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:45.686407 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.686341 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-workload-certs" pod "1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" (UID: "1fb55a71-37cf-4f12-bb4e-6edef8a3db2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:45.688617 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.687079 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:45.688617 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.687104 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:45.688617 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.687117 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:45.688617 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.687132 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:45.688617 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.687144 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:45.688617 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.687435 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-envoy" pod "1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" (UID: "1fb55a71-37cf-4f12-bb4e-6edef8a3db2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:45.689309 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.689265 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-podinfo" pod "1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" (UID: "1fb55a71-37cf-4f12-bb4e-6edef8a3db2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:00:45.690001 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.689975 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-kube-api-access-2ng84" pod "1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" (UID: "1fb55a71-37cf-4f12-bb4e-6edef8a3db2e"). InnerVolumeSpecName "kube-api-access-2ng84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:45.690087 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.690065 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-token" pod "1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" (UID: "1fb55a71-37cf-4f12-bb4e-6edef8a3db2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:45.789093 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.789050 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ng84\" (UniqueName: \"kubernetes.io/projected/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-kube-api-access-2ng84\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:45.789093 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.789081 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:45.789093 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.789092 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:45.789093 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.789101 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:45.831874 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.831770 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" podUID="abcec173-ab52-4006-b573-f5917ac603c2" containerName="main" containerID="cri-o://f8f87679bf301a73988914748e888103075f1e3080753c78f4b17a88ce06d2b1" gracePeriod=45 Nov 03 17:00:45.904261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.904224 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-7zkln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:00:45.904452 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:45.904304 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" podUID="036fe6c6-378c-42f3-87cc-91315033ea59" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:00:46.102729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.102706 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 17:00:46.193177 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.193132 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-model-cache\") pod \"abcec173-ab52-4006-b573-f5917ac603c2\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " Nov 03 17:00:46.193383 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.193214 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abcec173-ab52-4006-b573-f5917ac603c2-kube-api-access-b9cbz\" (UniqueName: \"kubernetes.io/projected/abcec173-ab52-4006-b573-f5917ac603c2-kube-api-access-b9cbz\") pod \"abcec173-ab52-4006-b573-f5917ac603c2\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " Nov 03 17:00:46.193383 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.193251 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-home\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-home\") pod \"abcec173-ab52-4006-b573-f5917ac603c2\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " Nov 03 17:00:46.193383 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.193338 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-tmp-dir\") pod \"abcec173-ab52-4006-b573-f5917ac603c2\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " Nov 03 17:00:46.193575 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.193403 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-model-cache" pod "abcec173-ab52-4006-b573-f5917ac603c2" (UID: "abcec173-ab52-4006-b573-f5917ac603c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:46.193575 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.193470 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-dshm\") pod \"abcec173-ab52-4006-b573-f5917ac603c2\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " Nov 03 17:00:46.193575 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.193516 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abcec173-ab52-4006-b573-f5917ac603c2-tls-certs\" (UniqueName: \"kubernetes.io/secret/abcec173-ab52-4006-b573-f5917ac603c2-tls-certs\") pod \"abcec173-ab52-4006-b573-f5917ac603c2\" (UID: \"abcec173-ab52-4006-b573-f5917ac603c2\") " Nov 03 17:00:46.193575 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.193568 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-home" pod "abcec173-ab52-4006-b573-f5917ac603c2" (UID: "abcec173-ab52-4006-b573-f5917ac603c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:46.193834 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.193725 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-tmp-dir" pod "abcec173-ab52-4006-b573-f5917ac603c2" (UID: "abcec173-ab52-4006-b573-f5917ac603c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:46.194168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.194145 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:46.194168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.194167 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:46.194168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.194181 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:46.195946 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.195924 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-dshm" pod "abcec173-ab52-4006-b573-f5917ac603c2" (UID: "abcec173-ab52-4006-b573-f5917ac603c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:46.196329 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.196207 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abcec173-ab52-4006-b573-f5917ac603c2-tls-certs" pod "abcec173-ab52-4006-b573-f5917ac603c2" (UID: "abcec173-ab52-4006-b573-f5917ac603c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:00:46.196329 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.196283 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abcec173-ab52-4006-b573-f5917ac603c2-kube-api-access-b9cbz" pod "abcec173-ab52-4006-b573-f5917ac603c2" (UID: "abcec173-ab52-4006-b573-f5917ac603c2"). InnerVolumeSpecName "kube-api-access-b9cbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:46.295293 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.295256 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9cbz\" (UniqueName: \"kubernetes.io/projected/abcec173-ab52-4006-b573-f5917ac603c2-kube-api-access-b9cbz\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:46.295293 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.295290 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abcec173-ab52-4006-b573-f5917ac603c2-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:46.295557 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.295303 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abcec173-ab52-4006-b573-f5917ac603c2-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:46.487790 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.487647 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" Nov 03 17:00:46.487790 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.487654 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc" event={"ID":"1fb55a71-37cf-4f12-bb4e-6edef8a3db2e","Type":"ContainerDied","Data":"b4bb20628f0f54a74346e5eeafe5c0ad52dd66768df2a8f86a3bbc635c5bf271"} Nov 03 17:00:46.487790 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.487741 4947 scope.go:122] "RemoveContainer" containerID="52aff2feb0520554e0777b6ae637b85f0b2abea2e89aa85b6ea4d5c2329a2384" Nov 03 17:00:46.489337 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.489253 4947 generic.go:356] "Generic (PLEG): container finished" podID="abcec173-ab52-4006-b573-f5917ac603c2" containerID="f8f87679bf301a73988914748e888103075f1e3080753c78f4b17a88ce06d2b1" exitCode=0 Nov 03 17:00:46.489337 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.489289 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" event={"ID":"abcec173-ab52-4006-b573-f5917ac603c2","Type":"ContainerDied","Data":"f8f87679bf301a73988914748e888103075f1e3080753c78f4b17a88ce06d2b1"} Nov 03 17:00:46.489337 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.489309 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" Nov 03 17:00:46.489337 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.489328 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj" event={"ID":"abcec173-ab52-4006-b573-f5917ac603c2","Type":"ContainerDied","Data":"d039bd707e323d7dbdfb61d9286b3f505ae9839aee4e78670c3c83cd186703ea"} Nov 03 17:00:46.499569 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.499520 4947 scope.go:122] "RemoveContainer" containerID="f8f87679bf301a73988914748e888103075f1e3080753c78f4b17a88ce06d2b1" Nov 03 17:00:46.510775 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.510759 4947 scope.go:122] "RemoveContainer" containerID="f8f87679bf301a73988914748e888103075f1e3080753c78f4b17a88ce06d2b1" Nov 03 17:00:46.511054 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:00:46.511036 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8f87679bf301a73988914748e888103075f1e3080753c78f4b17a88ce06d2b1\": container with ID starting with f8f87679bf301a73988914748e888103075f1e3080753c78f4b17a88ce06d2b1 not found: ID does not exist" containerID="f8f87679bf301a73988914748e888103075f1e3080753c78f4b17a88ce06d2b1" Nov 03 17:00:46.511120 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.511063 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8f87679bf301a73988914748e888103075f1e3080753c78f4b17a88ce06d2b1"} err="failed to get container status \"f8f87679bf301a73988914748e888103075f1e3080753c78f4b17a88ce06d2b1\": rpc error: code = NotFound desc = could not find container \"f8f87679bf301a73988914748e888103075f1e3080753c78f4b17a88ce06d2b1\": container with ID starting with f8f87679bf301a73988914748e888103075f1e3080753c78f4b17a88ce06d2b1 not found: ID does not exist" Nov 03 17:00:46.547769 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.547731 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc"] Nov 03 17:00:46.551155 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.551118 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-8k8bc"] Nov 03 17:00:46.566531 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.566496 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" path="/var/lib/kubelet/pods/1fb55a71-37cf-4f12-bb4e-6edef8a3db2e/volumes" Nov 03 17:00:46.586466 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.586439 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj"] Nov 03 17:00:46.593609 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:46.593585 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3b6f35/auth-test-model-3b6f35-kserve-795fd94d65-cvlbj"] Nov 03 17:00:48.568012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:48.567970 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abcec173-ab52-4006-b573-f5917ac603c2" path="/var/lib/kubelet/pods/abcec173-ab52-4006-b573-f5917ac603c2/volumes" Nov 03 17:00:49.234510 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.234435 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 17:00:49.363649 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.363611 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-credential-socket\") pod \"036fe6c6-378c-42f3-87cc-91315033ea59\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " Nov 03 17:00:49.363884 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.363784 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036fe6c6-378c-42f3-87cc-91315033ea59-kube-api-access-pzqqf\" (UniqueName: \"kubernetes.io/projected/036fe6c6-378c-42f3-87cc-91315033ea59-kube-api-access-pzqqf\") pod \"036fe6c6-378c-42f3-87cc-91315033ea59\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " Nov 03 17:00:49.363884 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.363805 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-workload-socket\") pod \"036fe6c6-378c-42f3-87cc-91315033ea59\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " Nov 03 17:00:49.364019 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.363886 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/036fe6c6-378c-42f3-87cc-91315033ea59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036fe6c6-378c-42f3-87cc-91315033ea59-istiod-ca-cert\") pod \"036fe6c6-378c-42f3-87cc-91315033ea59\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " Nov 03 17:00:49.364019 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.363934 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-istio-envoy\") pod \"036fe6c6-378c-42f3-87cc-91315033ea59\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " Nov 03 17:00:49.364019 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.363980 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-istio-data\") pod \"036fe6c6-378c-42f3-87cc-91315033ea59\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " Nov 03 17:00:49.364019 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.363988 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-credential-socket" pod "036fe6c6-378c-42f3-87cc-91315033ea59" (UID: "036fe6c6-378c-42f3-87cc-91315033ea59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:49.364196 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.364073 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-workload-certs\") pod \"036fe6c6-378c-42f3-87cc-91315033ea59\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " Nov 03 17:00:49.364196 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.364125 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/036fe6c6-378c-42f3-87cc-91315033ea59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036fe6c6-378c-42f3-87cc-91315033ea59-istio-podinfo\") pod \"036fe6c6-378c-42f3-87cc-91315033ea59\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " Nov 03 17:00:49.364301 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.364196 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-workload-socket" pod "036fe6c6-378c-42f3-87cc-91315033ea59" (UID: "036fe6c6-378c-42f3-87cc-91315033ea59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:49.364301 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.364202 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036fe6c6-378c-42f3-87cc-91315033ea59-istio-token\" (UniqueName: \"kubernetes.io/projected/036fe6c6-378c-42f3-87cc-91315033ea59-istio-token\") pod \"036fe6c6-378c-42f3-87cc-91315033ea59\" (UID: \"036fe6c6-378c-42f3-87cc-91315033ea59\") " Nov 03 17:00:49.364410 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.364333 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-istio-data" pod "036fe6c6-378c-42f3-87cc-91315033ea59" (UID: "036fe6c6-378c-42f3-87cc-91315033ea59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:49.364410 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.364357 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/036fe6c6-378c-42f3-87cc-91315033ea59-istiod-ca-cert" pod "036fe6c6-378c-42f3-87cc-91315033ea59" (UID: "036fe6c6-378c-42f3-87cc-91315033ea59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:49.364410 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.364394 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-workload-certs" pod "036fe6c6-378c-42f3-87cc-91315033ea59" (UID: "036fe6c6-378c-42f3-87cc-91315033ea59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:49.365180 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.365151 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:49.365298 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.365184 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:49.365298 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.365201 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:49.365298 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.365215 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036fe6c6-378c-42f3-87cc-91315033ea59-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:49.365298 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.365228 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:49.366312 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.366289 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036fe6c6-378c-42f3-87cc-91315033ea59-kube-api-access-pzqqf" pod "036fe6c6-378c-42f3-87cc-91315033ea59" (UID: "036fe6c6-378c-42f3-87cc-91315033ea59"). InnerVolumeSpecName "kube-api-access-pzqqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:49.366399 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.366311 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-istio-envoy" pod "036fe6c6-378c-42f3-87cc-91315033ea59" (UID: "036fe6c6-378c-42f3-87cc-91315033ea59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:49.366492 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.366476 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/036fe6c6-378c-42f3-87cc-91315033ea59-istio-podinfo" pod "036fe6c6-378c-42f3-87cc-91315033ea59" (UID: "036fe6c6-378c-42f3-87cc-91315033ea59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:00:49.366593 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.366577 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036fe6c6-378c-42f3-87cc-91315033ea59-istio-token" pod "036fe6c6-378c-42f3-87cc-91315033ea59" (UID: "036fe6c6-378c-42f3-87cc-91315033ea59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:49.466917 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.466876 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzqqf\" (UniqueName: \"kubernetes.io/projected/036fe6c6-378c-42f3-87cc-91315033ea59-kube-api-access-pzqqf\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:49.466917 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.466910 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036fe6c6-378c-42f3-87cc-91315033ea59-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:49.466917 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.466918 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036fe6c6-378c-42f3-87cc-91315033ea59-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:49.466917 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.466928 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036fe6c6-378c-42f3-87cc-91315033ea59-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:49.513416 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.513383 4947 generic.go:356] "Generic (PLEG): container finished" podID="036fe6c6-378c-42f3-87cc-91315033ea59" containerID="01426ee92fff2e74418abcd33b6851b12a1c7f0bdfa8ff416a86576178d0059b" exitCode=0 Nov 03 17:00:49.513595 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.513428 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" Nov 03 17:00:49.513595 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.513469 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" event={"ID":"036fe6c6-378c-42f3-87cc-91315033ea59","Type":"ContainerDied","Data":"01426ee92fff2e74418abcd33b6851b12a1c7f0bdfa8ff416a86576178d0059b"} Nov 03 17:00:49.513595 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.513521 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln" event={"ID":"036fe6c6-378c-42f3-87cc-91315033ea59","Type":"ContainerDied","Data":"8a0a1809a0a147372c8975fb16f26491921e2142c214cb86d7bfec689a41e592"} Nov 03 17:00:49.513595 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.513544 4947 scope.go:122] "RemoveContainer" containerID="01426ee92fff2e74418abcd33b6851b12a1c7f0bdfa8ff416a86576178d0059b" Nov 03 17:00:49.525026 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.524999 4947 scope.go:122] "RemoveContainer" containerID="01426ee92fff2e74418abcd33b6851b12a1c7f0bdfa8ff416a86576178d0059b" Nov 03 17:00:49.525347 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:00:49.525331 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01426ee92fff2e74418abcd33b6851b12a1c7f0bdfa8ff416a86576178d0059b\": container with ID starting with 01426ee92fff2e74418abcd33b6851b12a1c7f0bdfa8ff416a86576178d0059b not found: ID does not exist" containerID="01426ee92fff2e74418abcd33b6851b12a1c7f0bdfa8ff416a86576178d0059b" Nov 03 17:00:49.525413 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.525357 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01426ee92fff2e74418abcd33b6851b12a1c7f0bdfa8ff416a86576178d0059b"} err="failed to get container status \"01426ee92fff2e74418abcd33b6851b12a1c7f0bdfa8ff416a86576178d0059b\": rpc error: code = NotFound desc = could not find container \"01426ee92fff2e74418abcd33b6851b12a1c7f0bdfa8ff416a86576178d0059b\": container with ID starting with 01426ee92fff2e74418abcd33b6851b12a1c7f0bdfa8ff416a86576178d0059b not found: ID does not exist" Nov 03 17:00:49.571214 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.571164 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln"] Nov 03 17:00:49.575212 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:49.575180 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-7zkln"] Nov 03 17:00:50.311139 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:50.311094 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55"] Nov 03 17:00:50.311365 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:50.311338 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" podUID="d2a9d92e-90e9-41b4-9eba-1faf4b485843" containerName="istio-proxy" containerID="cri-o://1dc605142923274967868572266f044c86a88992288bf7c87039f36d09a67bb6" gracePeriod=30 Nov 03 17:00:50.567306 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:50.567224 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="036fe6c6-378c-42f3-87cc-91315033ea59" path="/var/lib/kubelet/pods/036fe6c6-378c-42f3-87cc-91315033ea59/volumes" Nov 03 17:00:50.795771 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:50.795732 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk"] Nov 03 17:00:55.623605 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.623576 4947 generic.go:356] "Generic (PLEG): container finished" podID="d2a9d92e-90e9-41b4-9eba-1faf4b485843" containerID="1dc605142923274967868572266f044c86a88992288bf7c87039f36d09a67bb6" exitCode=0 Nov 03 17:00:55.640967 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.623646 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" event={"ID":"d2a9d92e-90e9-41b4-9eba-1faf4b485843","Type":"ContainerDied","Data":"1dc605142923274967868572266f044c86a88992288bf7c87039f36d09a67bb6"} Nov 03 17:00:55.658273 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.658249 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 17:00:55.723851 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.723813 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-data\") pod \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " Nov 03 17:00:55.724164 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.723896 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istiod-ca-cert\") pod \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " Nov 03 17:00:55.724164 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.723972 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-envoy\") pod \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " Nov 03 17:00:55.724164 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.724011 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-workload-certs\") pod \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " Nov 03 17:00:55.724164 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.724048 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-credential-socket\") pod \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " Nov 03 17:00:55.724164 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.724105 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a9d92e-90e9-41b4-9eba-1faf4b485843-kube-api-access-q47xs\" (UniqueName: \"kubernetes.io/projected/d2a9d92e-90e9-41b4-9eba-1faf4b485843-kube-api-access-q47xs\") pod \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " Nov 03 17:00:55.724565 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.724168 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-podinfo\") pod \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " Nov 03 17:00:55.724565 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.724232 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-token\" (UniqueName: \"kubernetes.io/projected/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-token\") pod \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " Nov 03 17:00:55.724565 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.724359 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-workload-socket\") pod \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\" (UID: \"d2a9d92e-90e9-41b4-9eba-1faf4b485843\") " Nov 03 17:00:55.724899 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.724878 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-workload-certs" pod "d2a9d92e-90e9-41b4-9eba-1faf4b485843" (UID: "d2a9d92e-90e9-41b4-9eba-1faf4b485843"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:55.724992 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.724971 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:55.725061 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.725037 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-data" pod "d2a9d92e-90e9-41b4-9eba-1faf4b485843" (UID: "d2a9d92e-90e9-41b4-9eba-1faf4b485843"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:55.725265 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.725243 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-credential-socket" pod "d2a9d92e-90e9-41b4-9eba-1faf4b485843" (UID: "d2a9d92e-90e9-41b4-9eba-1faf4b485843"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:55.725559 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.725532 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istiod-ca-cert" pod "d2a9d92e-90e9-41b4-9eba-1faf4b485843" (UID: "d2a9d92e-90e9-41b4-9eba-1faf4b485843"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:00:55.734086 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.734060 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-envoy" pod "d2a9d92e-90e9-41b4-9eba-1faf4b485843" (UID: "d2a9d92e-90e9-41b4-9eba-1faf4b485843"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:55.734224 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.734091 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-podinfo" pod "d2a9d92e-90e9-41b4-9eba-1faf4b485843" (UID: "d2a9d92e-90e9-41b4-9eba-1faf4b485843"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:00:55.734343 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.734242 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-token" pod "d2a9d92e-90e9-41b4-9eba-1faf4b485843" (UID: "d2a9d92e-90e9-41b4-9eba-1faf4b485843"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:55.737807 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.737783 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-workload-socket" pod "d2a9d92e-90e9-41b4-9eba-1faf4b485843" (UID: "d2a9d92e-90e9-41b4-9eba-1faf4b485843"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:00:55.742110 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.741857 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a9d92e-90e9-41b4-9eba-1faf4b485843-kube-api-access-q47xs" pod "d2a9d92e-90e9-41b4-9eba-1faf4b485843" (UID: "d2a9d92e-90e9-41b4-9eba-1faf4b485843"). InnerVolumeSpecName "kube-api-access-q47xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:00:55.826813 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.826770 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:55.826813 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.826799 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:55.826813 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.826808 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:55.826813 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.826816 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:55.826813 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.826824 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:55.827178 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.826831 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2a9d92e-90e9-41b4-9eba-1faf4b485843-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:55.827178 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.826841 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q47xs\" (UniqueName: \"kubernetes.io/projected/d2a9d92e-90e9-41b4-9eba-1faf4b485843-kube-api-access-q47xs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:55.827178 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:55.826850 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2a9d92e-90e9-41b4-9eba-1faf4b485843-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:00:56.655925 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:56.655885 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" event={"ID":"d2a9d92e-90e9-41b4-9eba-1faf4b485843","Type":"ContainerDied","Data":"59d60effde11c315e9ba33c458886a203130bf92be69ddff4bed11b557bfd229"} Nov 03 17:00:56.675389 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:56.656537 4947 scope.go:122] "RemoveContainer" containerID="1dc605142923274967868572266f044c86a88992288bf7c87039f36d09a67bb6" Nov 03 17:00:56.675389 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:56.656731 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55" Nov 03 17:00:56.748244 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:56.748206 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55"] Nov 03 17:00:56.761519 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:56.761488 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-rsp55"] Nov 03 17:00:58.566906 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:58.566875 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a9d92e-90e9-41b4-9eba-1faf4b485843" path="/var/lib/kubelet/pods/d2a9d92e-90e9-41b4-9eba-1faf4b485843/volumes" Nov 03 17:00:59.367593 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.367545 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm"] Nov 03 17:00:59.368275 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368258 4947 cpu_manager.go:414] "containerMap: removing container" podUID="9fb5522f-9240-4c7b-bc72-4fa24c5e1b99" containerName="cleanup" Nov 03 17:00:59.368333 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368277 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb5522f-9240-4c7b-bc72-4fa24c5e1b99" containerName="cleanup" Nov 03 17:00:59.368333 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368287 4947 cpu_manager.go:414] "containerMap: removing container" podUID="9fb5522f-9240-4c7b-bc72-4fa24c5e1b99" containerName="cleanup" Nov 03 17:00:59.368333 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368292 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb5522f-9240-4c7b-bc72-4fa24c5e1b99" containerName="cleanup" Nov 03 17:00:59.368333 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368299 4947 cpu_manager.go:414] "containerMap: removing container" podUID="d2a9d92e-90e9-41b4-9eba-1faf4b485843" containerName="istio-proxy" Nov 03 17:00:59.368333 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368305 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a9d92e-90e9-41b4-9eba-1faf4b485843" containerName="istio-proxy" Nov 03 17:00:59.368333 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368315 4947 cpu_manager.go:414] "containerMap: removing container" podUID="abcec173-ab52-4006-b573-f5917ac603c2" containerName="main" Nov 03 17:00:59.368333 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368319 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abcec173-ab52-4006-b573-f5917ac603c2" containerName="main" Nov 03 17:00:59.368549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368346 4947 cpu_manager.go:414] "containerMap: removing container" podUID="036fe6c6-378c-42f3-87cc-91315033ea59" containerName="istio-proxy" Nov 03 17:00:59.368549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368351 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="036fe6c6-378c-42f3-87cc-91315033ea59" containerName="istio-proxy" Nov 03 17:00:59.368549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368365 4947 cpu_manager.go:414] "containerMap: removing container" podUID="1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" containerName="istio-proxy" Nov 03 17:00:59.368549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368370 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" containerName="istio-proxy" Nov 03 17:00:59.368549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368379 4947 cpu_manager.go:414] "containerMap: removing container" podUID="b720f231-ec3b-4e94-bbf5-8920f8976a9e" containerName="istio-proxy" Nov 03 17:00:59.368549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368383 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b720f231-ec3b-4e94-bbf5-8920f8976a9e" containerName="istio-proxy" Nov 03 17:00:59.368549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368470 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="b720f231-ec3b-4e94-bbf5-8920f8976a9e" containerName="istio-proxy" Nov 03 17:00:59.368549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368482 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb5522f-9240-4c7b-bc72-4fa24c5e1b99" containerName="cleanup" Nov 03 17:00:59.368549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368488 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb5522f-9240-4c7b-bc72-4fa24c5e1b99" containerName="cleanup" Nov 03 17:00:59.368549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368494 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a9d92e-90e9-41b4-9eba-1faf4b485843" containerName="istio-proxy" Nov 03 17:00:59.368549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368498 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="abcec173-ab52-4006-b573-f5917ac603c2" containerName="main" Nov 03 17:00:59.368549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368505 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce9ff523-dcb8-42d3-b7b4-dc2734a6ca47" containerName="cleanup" Nov 03 17:00:59.368549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368510 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb5522f-9240-4c7b-bc72-4fa24c5e1b99" containerName="cleanup" Nov 03 17:00:59.368549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368519 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="036fe6c6-378c-42f3-87cc-91315033ea59" containerName="istio-proxy" Nov 03 17:00:59.368549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.368526 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb55a71-37cf-4f12-bb4e-6edef8a3db2e" containerName="istio-proxy" Nov 03 17:00:59.373466 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.373444 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" Nov 03 17:00:59.392282 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.392261 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-1f83c0\"" Nov 03 17:00:59.392584 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.392563 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-1f83c0-dockercfg-q69cp\"" Nov 03 17:00:59.410384 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.410360 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm"] Nov 03 17:00:59.500725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.494535 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65d3f051-43e5-43a7-a879-5a14425498db-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm\" (UID: \"65d3f051-43e5-43a7-a879-5a14425498db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" Nov 03 17:00:59.500725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.494616 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fxnb\" (UniqueName: \"kubernetes.io/projected/65d3f051-43e5-43a7-a879-5a14425498db-kube-api-access-4fxnb\") pod \"payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm\" (UID: \"65d3f051-43e5-43a7-a879-5a14425498db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" Nov 03 17:00:59.609587 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.603163 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65d3f051-43e5-43a7-a879-5a14425498db-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm\" (UID: \"65d3f051-43e5-43a7-a879-5a14425498db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" Nov 03 17:00:59.609587 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.603254 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fxnb\" (UniqueName: \"kubernetes.io/projected/65d3f051-43e5-43a7-a879-5a14425498db-kube-api-access-4fxnb\") pod \"payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm\" (UID: \"65d3f051-43e5-43a7-a879-5a14425498db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" Nov 03 17:00:59.609587 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.604333 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65d3f051-43e5-43a7-a879-5a14425498db-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm\" (UID: \"65d3f051-43e5-43a7-a879-5a14425498db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" Nov 03 17:00:59.615967 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.615937 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l"] Nov 03 17:00:59.618040 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.617971 4947 cpu_manager.go:414] "containerMap: removing container" podUID="9fb5522f-9240-4c7b-bc72-4fa24c5e1b99" containerName="cleanup" Nov 03 17:00:59.618209 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.618192 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb5522f-9240-4c7b-bc72-4fa24c5e1b99" containerName="cleanup" Nov 03 17:00:59.644987 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.644958 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l"] Nov 03 17:00:59.645205 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.645060 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" Nov 03 17:00:59.648938 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.648911 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fxnb\" (UniqueName: \"kubernetes.io/projected/65d3f051-43e5-43a7-a879-5a14425498db-kube-api-access-4fxnb\") pod \"payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm\" (UID: \"65d3f051-43e5-43a7-a879-5a14425498db\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" Nov 03 17:00:59.686749 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.686692 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" podUID="fb3ce8a7-ce9a-459d-8580-5fe627e79065" containerName="main" containerID="cri-o://aa17bd96bbe11b1f8ee5f232671796e1cb88c1992e82ae2a9754c31970b6fd95" gracePeriod=45 Nov 03 17:00:59.691056 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.691026 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" Nov 03 17:00:59.705156 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.705128 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6fe77c5-ca27-418f-971d-842b755b3e5a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l\" (UID: \"d6fe77c5-ca27-418f-971d-842b755b3e5a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" Nov 03 17:00:59.705368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.705345 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twrf4\" (UniqueName: \"kubernetes.io/projected/d6fe77c5-ca27-418f-971d-842b755b3e5a-kube-api-access-twrf4\") pod \"payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l\" (UID: \"d6fe77c5-ca27-418f-971d-842b755b3e5a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" Nov 03 17:00:59.807294 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.807255 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twrf4\" (UniqueName: \"kubernetes.io/projected/d6fe77c5-ca27-418f-971d-842b755b3e5a-kube-api-access-twrf4\") pod \"payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l\" (UID: \"d6fe77c5-ca27-418f-971d-842b755b3e5a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" Nov 03 17:00:59.807482 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.807422 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6fe77c5-ca27-418f-971d-842b755b3e5a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l\" (UID: \"d6fe77c5-ca27-418f-971d-842b755b3e5a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" Nov 03 17:00:59.808567 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.808507 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6fe77c5-ca27-418f-971d-842b755b3e5a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l\" (UID: \"d6fe77c5-ca27-418f-971d-842b755b3e5a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" Nov 03 17:00:59.837634 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.837601 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twrf4\" (UniqueName: \"kubernetes.io/projected/d6fe77c5-ca27-418f-971d-842b755b3e5a-kube-api-access-twrf4\") pod \"payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l\" (UID: \"d6fe77c5-ca27-418f-971d-842b755b3e5a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" Nov 03 17:00:59.965779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.965751 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 17:00:59.974449 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:00:59.974420 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" Nov 03 17:01:00.010048 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.010017 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-home\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-home\") pod \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " Nov 03 17:01:00.010214 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.010064 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb3ce8a7-ce9a-459d-8580-5fe627e79065-kube-api-access-n4sqv\" (UniqueName: \"kubernetes.io/projected/fb3ce8a7-ce9a-459d-8580-5fe627e79065-kube-api-access-n4sqv\") pod \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " Nov 03 17:01:00.010214 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.010117 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb3ce8a7-ce9a-459d-8580-5fe627e79065-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb3ce8a7-ce9a-459d-8580-5fe627e79065-tls-certs\") pod \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " Nov 03 17:01:00.010214 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:01:00.010133 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d3f051_43e5_43a7_a879_5a14425498db.slice/crio-bf4ac4bfe0082532d1a3fd726ef0d1051622521e5d9f3006698e920e5697a48f WatchSource:0}: Error finding container bf4ac4bfe0082532d1a3fd726ef0d1051622521e5d9f3006698e920e5697a48f: Status 404 returned error can't find the container with id bf4ac4bfe0082532d1a3fd726ef0d1051622521e5d9f3006698e920e5697a48f Nov 03 17:01:00.010214 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.010196 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-dshm\") pod \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " Nov 03 17:01:00.010447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.010283 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-model-cache\") pod \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " Nov 03 17:01:00.010447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.010308 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-home" pod "fb3ce8a7-ce9a-459d-8580-5fe627e79065" (UID: "fb3ce8a7-ce9a-459d-8580-5fe627e79065"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:00.010447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.010349 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-tmp-dir\") pod \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\" (UID: \"fb3ce8a7-ce9a-459d-8580-5fe627e79065\") " Nov 03 17:01:00.011101 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.010864 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:01:00.011101 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.010881 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-model-cache" pod "fb3ce8a7-ce9a-459d-8580-5fe627e79065" (UID: "fb3ce8a7-ce9a-459d-8580-5fe627e79065"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:00.011401 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.011126 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-tmp-dir" pod "fb3ce8a7-ce9a-459d-8580-5fe627e79065" (UID: "fb3ce8a7-ce9a-459d-8580-5fe627e79065"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:00.011880 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.011849 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm"] Nov 03 17:01:00.013219 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.013172 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-dshm" pod "fb3ce8a7-ce9a-459d-8580-5fe627e79065" (UID: "fb3ce8a7-ce9a-459d-8580-5fe627e79065"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:00.013362 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.013266 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb3ce8a7-ce9a-459d-8580-5fe627e79065-tls-certs" pod "fb3ce8a7-ce9a-459d-8580-5fe627e79065" (UID: "fb3ce8a7-ce9a-459d-8580-5fe627e79065"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:01:00.013459 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.013438 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3ce8a7-ce9a-459d-8580-5fe627e79065-kube-api-access-n4sqv" pod "fb3ce8a7-ce9a-459d-8580-5fe627e79065" (UID: "fb3ce8a7-ce9a-459d-8580-5fe627e79065"). InnerVolumeSpecName "kube-api-access-n4sqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:00.112270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.112200 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb3ce8a7-ce9a-459d-8580-5fe627e79065-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:01:00.112270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.112224 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:01:00.112270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.112232 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:01:00.112270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.112239 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb3ce8a7-ce9a-459d-8580-5fe627e79065-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:01:00.112270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.112247 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4sqv\" (UniqueName: \"kubernetes.io/projected/fb3ce8a7-ce9a-459d-8580-5fe627e79065-kube-api-access-n4sqv\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:01:00.137054 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:01:00.137014 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:01:00.295874 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.295848 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l"] Nov 03 17:01:00.297844 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:01:00.297816 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6fe77c5_ca27_418f_971d_842b755b3e5a.slice/crio-0127231d5b3c56669230d0045e96ce5738cd1d55261cf09c50c4e26978342750 WatchSource:0}: Error finding container 0127231d5b3c56669230d0045e96ce5738cd1d55261cf09c50c4e26978342750: Status 404 returned error can't find the container with id 0127231d5b3c56669230d0045e96ce5738cd1d55261cf09c50c4e26978342750 Nov 03 17:01:00.684415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.684376 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" event={"ID":"65d3f051-43e5-43a7-a879-5a14425498db","Type":"ContainerStarted","Data":"cf6cdce8dab586b2a17cd0c58f46001d1109aef4b46358e6819623377d83d0eb"} Nov 03 17:01:00.684879 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.684438 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" event={"ID":"65d3f051-43e5-43a7-a879-5a14425498db","Type":"ContainerStarted","Data":"bf4ac4bfe0082532d1a3fd726ef0d1051622521e5d9f3006698e920e5697a48f"} Nov 03 17:01:00.684879 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.684463 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" Nov 03 17:01:00.685875 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.685846 4947 generic.go:356] "Generic (PLEG): container finished" podID="fb3ce8a7-ce9a-459d-8580-5fe627e79065" containerID="aa17bd96bbe11b1f8ee5f232671796e1cb88c1992e82ae2a9754c31970b6fd95" exitCode=0 Nov 03 17:01:00.685988 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.685884 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" Nov 03 17:01:00.685988 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.685933 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" event={"ID":"fb3ce8a7-ce9a-459d-8580-5fe627e79065","Type":"ContainerDied","Data":"aa17bd96bbe11b1f8ee5f232671796e1cb88c1992e82ae2a9754c31970b6fd95"} Nov 03 17:01:00.685988 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.685965 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh" event={"ID":"fb3ce8a7-ce9a-459d-8580-5fe627e79065","Type":"ContainerDied","Data":"67bc0d2981936fc5a509ab6ea11d015887583f4db2c7e18c28f876530fb9ffc5"} Nov 03 17:01:00.685988 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.685982 4947 scope.go:122] "RemoveContainer" containerID="aa17bd96bbe11b1f8ee5f232671796e1cb88c1992e82ae2a9754c31970b6fd95" Nov 03 17:01:00.687948 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.687924 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" event={"ID":"d6fe77c5-ca27-418f-971d-842b755b3e5a","Type":"ContainerStarted","Data":"f39f898350221526e5db4fe81c5da892965d7e4d0e89dd07535d32c83b932620"} Nov 03 17:01:00.688086 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.687971 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" event={"ID":"d6fe77c5-ca27-418f-971d-842b755b3e5a","Type":"ContainerStarted","Data":"0127231d5b3c56669230d0045e96ce5738cd1d55261cf09c50c4e26978342750"} Nov 03 17:01:00.688157 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.688139 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" Nov 03 17:01:00.696485 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.696470 4947 scope.go:122] "RemoveContainer" containerID="aa17bd96bbe11b1f8ee5f232671796e1cb88c1992e82ae2a9754c31970b6fd95" Nov 03 17:01:00.696756 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:01:00.696732 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa17bd96bbe11b1f8ee5f232671796e1cb88c1992e82ae2a9754c31970b6fd95\": container with ID starting with aa17bd96bbe11b1f8ee5f232671796e1cb88c1992e82ae2a9754c31970b6fd95 not found: ID does not exist" containerID="aa17bd96bbe11b1f8ee5f232671796e1cb88c1992e82ae2a9754c31970b6fd95" Nov 03 17:01:00.696822 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.696765 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa17bd96bbe11b1f8ee5f232671796e1cb88c1992e82ae2a9754c31970b6fd95"} err="failed to get container status \"aa17bd96bbe11b1f8ee5f232671796e1cb88c1992e82ae2a9754c31970b6fd95\": rpc error: code = NotFound desc = could not find container \"aa17bd96bbe11b1f8ee5f232671796e1cb88c1992e82ae2a9754c31970b6fd95\": container with ID starting with aa17bd96bbe11b1f8ee5f232671796e1cb88c1992e82ae2a9754c31970b6fd95 not found: ID does not exist" Nov 03 17:01:00.739583 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.735642 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" podStartSLOduration=1.735625398 podStartE2EDuration="1.735625398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:00.733383922 +0000 UTC m=+2804.322383891" watchObservedRunningTime="2026-11-03 17:01:00.735625398 +0000 UTC m=+2804.324625368" Nov 03 17:01:00.786375 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.786317 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" podStartSLOduration=1.786302166 podStartE2EDuration="1.786302166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:00.783334799 +0000 UTC m=+2804.372334768" watchObservedRunningTime="2026-11-03 17:01:00.786302166 +0000 UTC m=+2804.375302155" Nov 03 17:01:00.823339 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.823309 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh"] Nov 03 17:01:00.827841 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:00.827778 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-9a80f0/auth-test-model-9a80f0-kserve-549b57c99c-c8krh"] Nov 03 17:01:02.567175 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:02.567147 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb3ce8a7-ce9a-459d-8580-5fe627e79065" path="/var/lib/kubelet/pods/fb3ce8a7-ce9a-459d-8580-5fe627e79065/volumes" Nov 03 17:01:06.550283 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.550235 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" podUID="ef66bd84-620e-40a2-9c81-489eb827e636" containerName="main" containerID="cri-o://a54389217dca4f94beb5a86f71f970751c9e422175a9ec2e8c17f901f1f0b898" gracePeriod=45 Nov 03 17:01:06.738199 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.738162 4947 generic.go:356] "Generic (PLEG): container finished" podID="ef66bd84-620e-40a2-9c81-489eb827e636" containerID="a54389217dca4f94beb5a86f71f970751c9e422175a9ec2e8c17f901f1f0b898" exitCode=0 Nov 03 17:01:06.739324 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.739296 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" event={"ID":"ef66bd84-620e-40a2-9c81-489eb827e636","Type":"ContainerDied","Data":"a54389217dca4f94beb5a86f71f970751c9e422175a9ec2e8c17f901f1f0b898"} Nov 03 17:01:06.823844 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.823818 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 17:01:06.910285 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.910247 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-home\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-home\") pod \"ef66bd84-620e-40a2-9c81-489eb827e636\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " Nov 03 17:01:06.910465 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.910387 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef66bd84-620e-40a2-9c81-489eb827e636-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef66bd84-620e-40a2-9c81-489eb827e636-tls-certs\") pod \"ef66bd84-620e-40a2-9c81-489eb827e636\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " Nov 03 17:01:06.910465 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.910448 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-dshm\") pod \"ef66bd84-620e-40a2-9c81-489eb827e636\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " Nov 03 17:01:06.910586 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.910496 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-model-cache\") pod \"ef66bd84-620e-40a2-9c81-489eb827e636\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " Nov 03 17:01:06.910586 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.910499 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-home" pod "ef66bd84-620e-40a2-9c81-489eb827e636" (UID: "ef66bd84-620e-40a2-9c81-489eb827e636"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:06.910690 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.910676 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-tmp-dir\") pod \"ef66bd84-620e-40a2-9c81-489eb827e636\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " Nov 03 17:01:06.910802 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.910775 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-model-cache" pod "ef66bd84-620e-40a2-9c81-489eb827e636" (UID: "ef66bd84-620e-40a2-9c81-489eb827e636"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:06.910802 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.910790 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef66bd84-620e-40a2-9c81-489eb827e636-kube-api-access-hg8rp\" (UniqueName: \"kubernetes.io/projected/ef66bd84-620e-40a2-9c81-489eb827e636-kube-api-access-hg8rp\") pod \"ef66bd84-620e-40a2-9c81-489eb827e636\" (UID: \"ef66bd84-620e-40a2-9c81-489eb827e636\") " Nov 03 17:01:06.912552 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.910964 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-tmp-dir" pod "ef66bd84-620e-40a2-9c81-489eb827e636" (UID: "ef66bd84-620e-40a2-9c81-489eb827e636"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:06.912552 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.911527 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:01:06.912552 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.911550 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:01:06.912552 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.911565 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:01:06.913293 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.913270 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-dshm" pod "ef66bd84-620e-40a2-9c81-489eb827e636" (UID: "ef66bd84-620e-40a2-9c81-489eb827e636"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:01:06.917867 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.917844 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef66bd84-620e-40a2-9c81-489eb827e636-kube-api-access-hg8rp" pod "ef66bd84-620e-40a2-9c81-489eb827e636" (UID: "ef66bd84-620e-40a2-9c81-489eb827e636"). InnerVolumeSpecName "kube-api-access-hg8rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:01:06.922773 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:06.922755 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef66bd84-620e-40a2-9c81-489eb827e636-tls-certs" pod "ef66bd84-620e-40a2-9c81-489eb827e636" (UID: "ef66bd84-620e-40a2-9c81-489eb827e636"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:01:07.013089 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:07.013044 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg8rp\" (UniqueName: \"kubernetes.io/projected/ef66bd84-620e-40a2-9c81-489eb827e636-kube-api-access-hg8rp\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:01:07.013089 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:07.013084 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef66bd84-620e-40a2-9c81-489eb827e636-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:01:07.013089 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:07.013095 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef66bd84-620e-40a2-9c81-489eb827e636-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:01:07.755993 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:07.755960 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" Nov 03 17:01:07.755993 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:07.755975 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" event={"ID":"ef66bd84-620e-40a2-9c81-489eb827e636","Type":"ContainerDied","Data":"9bf8547f3c6eb66c6d4dcf5cbd3e31ff082832ad51b674ea01ce019c641a2f5e"} Nov 03 17:01:07.777958 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:07.756037 4947 scope.go:122] "RemoveContainer" containerID="a54389217dca4f94beb5a86f71f970751c9e422175a9ec2e8c17f901f1f0b898" Nov 03 17:01:07.813215 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:07.813190 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk"] Nov 03 17:01:07.815133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:07.815089 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk" podUID="ef66bd84-620e-40a2-9c81-489eb827e636" containerName="main" probeResult="failure" output="Get \"https://10.132.0.164:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:01:07.825655 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:07.825632 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-001802/auth-test-model-001802-kserve-68dcf67cbf-2wwtk"] Nov 03 17:01:08.566571 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:08.566530 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef66bd84-620e-40a2-9c81-489eb827e636" path="/var/lib/kubelet/pods/ef66bd84-620e-40a2-9c81-489eb827e636/volumes" Nov 03 17:01:11.698948 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:11.698920 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" Nov 03 17:01:11.699339 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:11.698989 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" Nov 03 17:01:14.958124 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:14.958092 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm"] Nov 03 17:01:14.958791 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:14.958777 4947 cpu_manager.go:414] "containerMap: removing container" podUID="ef66bd84-620e-40a2-9c81-489eb827e636" containerName="main" Nov 03 17:01:14.958791 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:14.958792 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef66bd84-620e-40a2-9c81-489eb827e636" containerName="main" Nov 03 17:01:14.958911 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:14.958833 4947 cpu_manager.go:414] "containerMap: removing container" podUID="fb3ce8a7-ce9a-459d-8580-5fe627e79065" containerName="main" Nov 03 17:01:14.958911 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:14.958838 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb3ce8a7-ce9a-459d-8580-5fe627e79065" containerName="main" Nov 03 17:01:14.958988 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:14.958931 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb3ce8a7-ce9a-459d-8580-5fe627e79065" containerName="main" Nov 03 17:01:14.958988 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:14.958944 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef66bd84-620e-40a2-9c81-489eb827e636" containerName="main" Nov 03 17:01:14.962674 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:14.962652 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" Nov 03 17:01:14.990868 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:14.990840 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-d6fcae-serving-cert\"" Nov 03 17:01:14.991884 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:14.991859 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm"] Nov 03 17:01:15.062251 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:15.062224 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm\" (UID: \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" Nov 03 17:01:15.062410 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:15.062384 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzlmd\" (UniqueName: \"kubernetes.io/projected/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-kube-api-access-bzlmd\") pod \"maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm\" (UID: \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" Nov 03 17:01:15.062471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:15.062456 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm\" (UID: \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" Nov 03 17:01:15.062535 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:15.062524 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-metrics-certs\") pod \"maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm\" (UID: \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" Nov 03 17:01:15.163313 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:15.163283 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzlmd\" (UniqueName: \"kubernetes.io/projected/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-kube-api-access-bzlmd\") pod \"maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm\" (UID: \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" Nov 03 17:01:15.163522 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:15.163354 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm\" (UID: \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" Nov 03 17:01:15.163522 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:15.163408 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-metrics-certs\") pod \"maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm\" (UID: \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" Nov 03 17:01:15.163522 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:15.163456 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm\" (UID: \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" Nov 03 17:01:15.164188 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:15.164168 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm\" (UID: \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" Nov 03 17:01:15.166021 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:15.166004 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm\" (UID: \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" Nov 03 17:01:15.166097 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:15.166076 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-metrics-certs\") pod \"maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm\" (UID: \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" Nov 03 17:01:15.189101 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:15.189079 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzlmd\" (UniqueName: \"kubernetes.io/projected/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-kube-api-access-bzlmd\") pod \"maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm\" (UID: \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" Nov 03 17:01:15.283184 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:15.283151 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" Nov 03 17:01:15.599882 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:15.599855 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm"] Nov 03 17:01:15.600619 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:01:15.600594 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f62b83b_5ad8_41aa_9e18_44cab99b41fd.slice/crio-a9959697fce88778dc79db527f0ba31a171d2299e4282e5e01f712f9b5817649 WatchSource:0}: Error finding container a9959697fce88778dc79db527f0ba31a171d2299e4282e5e01f712f9b5817649: Status 404 returned error can't find the container with id a9959697fce88778dc79db527f0ba31a171d2299e4282e5e01f712f9b5817649 Nov 03 17:01:15.819205 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:15.819170 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" event={"ID":"8f62b83b-5ad8-41aa-9e18-44cab99b41fd","Type":"ContainerStarted","Data":"a9959697fce88778dc79db527f0ba31a171d2299e4282e5e01f712f9b5817649"} Nov 03 17:01:16.828219 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:16.828186 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" event={"ID":"8f62b83b-5ad8-41aa-9e18-44cab99b41fd","Type":"ContainerStarted","Data":"2c38ccfe731250c06a13e85bc46d7459f0df6685b1dd2530eb277f760e2d42be"} Nov 03 17:01:16.828649 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:16.828312 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" Nov 03 17:01:16.877010 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:16.876955 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" podStartSLOduration=2.49533847 podStartE2EDuration="2.876940573s" totalImagesPullingTime="381.602103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:16.873646879 +0000 UTC m=+2820.462646846" watchObservedRunningTime="2026-11-03 17:01:16.876940573 +0000 UTC m=+2820.465940535" Nov 03 17:01:22.116933 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.116891 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2"] Nov 03 17:01:22.123023 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.122999 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.149046 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.149014 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2"] Nov 03 17:01:22.180917 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.180885 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-podinfo\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.181098 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.181012 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-workload-certs\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.181174 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.181119 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-token\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.181234 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.181218 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b50359-b597-46ac-9ca1-af75415e1cf1-istiod-ca-cert\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.181295 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.181271 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-data\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.181360 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.181346 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-envoy\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.181426 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.181383 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-credential-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.181426 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.181413 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpqwd\" (UniqueName: \"kubernetes.io/projected/e2b50359-b597-46ac-9ca1-af75415e1cf1-kube-api-access-qpqwd\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.181529 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.181448 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-workload-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.237720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.233732 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-e2d835-openshift-default-dockercfg-8f5zz\"" Nov 03 17:01:22.282615 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.282583 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-podinfo\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.282861 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.282642 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-workload-certs\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.282861 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.282708 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-token\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.282861 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.282746 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b50359-b597-46ac-9ca1-af75415e1cf1-istiod-ca-cert\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.282861 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.282800 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-data\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.282861 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.282845 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-envoy\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.283186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.282890 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-credential-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.283186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.282942 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpqwd\" (UniqueName: \"kubernetes.io/projected/e2b50359-b597-46ac-9ca1-af75415e1cf1-kube-api-access-qpqwd\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.283186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.283086 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-workload-certs\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.283186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.283132 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-workload-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.283373 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.283271 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-data\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.283587 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.283563 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-workload-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.283969 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.283672 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-credential-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.284168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.284143 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b50359-b597-46ac-9ca1-af75415e1cf1-istiod-ca-cert\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.285430 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.285405 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-envoy\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.285616 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.285601 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-podinfo\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.308354 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.308333 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-token\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.308443 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.308374 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpqwd\" (UniqueName: \"kubernetes.io/projected/e2b50359-b597-46ac-9ca1-af75415e1cf1-kube-api-access-qpqwd\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.439366 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.439279 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:22.771127 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.771091 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2"] Nov 03 17:01:22.772198 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:01:22.772167 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b50359_b597_46ac_9ca1_af75415e1cf1.slice/crio-ac1521be15b0b2f57ebb8fff60206e2b2f81a56461879954d495beaab363fdf8 WatchSource:0}: Error finding container ac1521be15b0b2f57ebb8fff60206e2b2f81a56461879954d495beaab363fdf8: Status 404 returned error can't find the container with id ac1521be15b0b2f57ebb8fff60206e2b2f81a56461879954d495beaab363fdf8 Nov 03 17:01:22.775998 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.775855 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:01:22.775998 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.775940 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:01:22.775998 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.775984 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:01:22.913402 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.908863 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" Nov 03 17:01:22.962078 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.960585 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh"] Nov 03 17:01:22.971981 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.971949 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" Nov 03 17:01:22.983783 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.975915 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh"] Nov 03 17:01:22.997549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.991281 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6046c2\"" Nov 03 17:01:22.997549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:22.992093 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-6046c2-dockercfg-jh629\"" Nov 03 17:01:23.003681 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:23.003652 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" event={"ID":"e2b50359-b597-46ac-9ca1-af75415e1cf1","Type":"ContainerStarted","Data":"ac1521be15b0b2f57ebb8fff60206e2b2f81a56461879954d495beaab363fdf8"} Nov 03 17:01:23.029842 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:23.029814 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f5d9a3-9c9e-404f-a572-7821d58bcb8d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh\" (UID: \"97f5d9a3-9c9e-404f-a572-7821d58bcb8d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" Nov 03 17:01:23.030134 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:23.030111 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsfd5\" (UniqueName: \"kubernetes.io/projected/97f5d9a3-9c9e-404f-a572-7821d58bcb8d-kube-api-access-nsfd5\") pod \"payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh\" (UID: \"97f5d9a3-9c9e-404f-a572-7821d58bcb8d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" Nov 03 17:01:23.132719 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:23.132254 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsfd5\" (UniqueName: \"kubernetes.io/projected/97f5d9a3-9c9e-404f-a572-7821d58bcb8d-kube-api-access-nsfd5\") pod \"payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh\" (UID: \"97f5d9a3-9c9e-404f-a572-7821d58bcb8d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" Nov 03 17:01:23.132719 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:23.132436 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f5d9a3-9c9e-404f-a572-7821d58bcb8d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh\" (UID: \"97f5d9a3-9c9e-404f-a572-7821d58bcb8d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" Nov 03 17:01:23.140011 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:23.137624 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f5d9a3-9c9e-404f-a572-7821d58bcb8d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh\" (UID: \"97f5d9a3-9c9e-404f-a572-7821d58bcb8d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" Nov 03 17:01:23.159129 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:23.159082 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsfd5\" (UniqueName: \"kubernetes.io/projected/97f5d9a3-9c9e-404f-a572-7821d58bcb8d-kube-api-access-nsfd5\") pod \"payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh\" (UID: \"97f5d9a3-9c9e-404f-a572-7821d58bcb8d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" Nov 03 17:01:23.303888 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:23.303797 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" Nov 03 17:01:23.619002 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:01:23.618973 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f5d9a3_9c9e_404f_a572_7821d58bcb8d.slice/crio-52fb7adc0376e2271ef291793319085a374e4f8bee29577d901c54ac93a66aa5 WatchSource:0}: Error finding container 52fb7adc0376e2271ef291793319085a374e4f8bee29577d901c54ac93a66aa5: Status 404 returned error can't find the container with id 52fb7adc0376e2271ef291793319085a374e4f8bee29577d901c54ac93a66aa5 Nov 03 17:01:23.619468 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:23.619443 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh"] Nov 03 17:01:24.031624 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:24.031584 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" event={"ID":"97f5d9a3-9c9e-404f-a572-7821d58bcb8d","Type":"ContainerStarted","Data":"023c332325aeab627e2aa1cb8ed66062893c8d9533457b02a8a53e6aabe13df5"} Nov 03 17:01:24.031848 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:24.031648 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" event={"ID":"97f5d9a3-9c9e-404f-a572-7821d58bcb8d","Type":"ContainerStarted","Data":"52fb7adc0376e2271ef291793319085a374e4f8bee29577d901c54ac93a66aa5"} Nov 03 17:01:24.031908 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:24.031871 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" Nov 03 17:01:24.033547 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:24.033524 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" event={"ID":"e2b50359-b597-46ac-9ca1-af75415e1cf1","Type":"ContainerStarted","Data":"caa492307a1a50d1f625a08b675f0cc1e9884d6dc1b4ebf4114e40b128ab1bec"} Nov 03 17:01:24.076336 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:24.076280 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" podStartSLOduration=2.076263934 podStartE2EDuration="2.076263934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:24.075624379 +0000 UTC m=+2827.664624348" watchObservedRunningTime="2026-11-03 17:01:24.076263934 +0000 UTC m=+2827.665263901" Nov 03 17:01:24.129654 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:24.129594 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" podStartSLOduration=2.129578981 podStartE2EDuration="2.129578981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:24.128675767 +0000 UTC m=+2827.717675736" watchObservedRunningTime="2026-11-03 17:01:24.129578981 +0000 UTC m=+2827.718578948" Nov 03 17:01:24.441795 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:24.441663 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:24.442276 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:24.441875 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 17:01:24.442276 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:24.441919 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" podUID="e2b50359-b597-46ac-9ca1-af75415e1cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 17:01:25.440421 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:25.440384 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 17:01:25.440650 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:25.440449 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" podUID="e2b50359-b597-46ac-9ca1-af75415e1cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 17:01:26.440454 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:26.440418 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 17:01:26.440922 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:26.440472 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" podUID="e2b50359-b597-46ac-9ca1-af75415e1cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 17:01:27.444786 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:27.443161 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 17:01:27.444786 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:27.443225 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" podUID="e2b50359-b597-46ac-9ca1-af75415e1cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 17:01:28.440782 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:28.440687 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body= Nov 03 17:01:28.440979 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:28.440807 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" podUID="e2b50359-b597-46ac-9ca1-af75415e1cf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" Nov 03 17:01:29.448278 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:29.448241 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:29.449229 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:29.449201 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:29.450523 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:29.450497 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:01:30.131018 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:01:30.130980 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:01:31.342687 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:31.342530 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d"] Nov 03 17:01:31.348824 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:31.348794 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:01:31.384741 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:31.378619 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-1f83c0-serving-cert\"" Nov 03 17:01:31.388127 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:31.388087 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d"] Nov 03 17:01:31.418598 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:31.418560 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd191dc-3ff4-4451-b93a-da68f606c84e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d\" (UID: \"afd191dc-3ff4-4451-b93a-da68f606c84e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:01:31.419086 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:31.419047 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9x8m\" (UniqueName: \"kubernetes.io/projected/afd191dc-3ff4-4451-b93a-da68f606c84e-kube-api-access-g9x8m\") pod \"maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d\" (UID: \"afd191dc-3ff4-4451-b93a-da68f606c84e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:01:31.419307 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:31.419287 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd191dc-3ff4-4451-b93a-da68f606c84e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d\" (UID: \"afd191dc-3ff4-4451-b93a-da68f606c84e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:01:31.419540 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:31.419520 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd191dc-3ff4-4451-b93a-da68f606c84e-metrics-certs\") pod \"maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d\" (UID: \"afd191dc-3ff4-4451-b93a-da68f606c84e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:01:31.520638 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:31.520601 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9x8m\" (UniqueName: \"kubernetes.io/projected/afd191dc-3ff4-4451-b93a-da68f606c84e-kube-api-access-g9x8m\") pod \"maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d\" (UID: \"afd191dc-3ff4-4451-b93a-da68f606c84e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:01:31.520901 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:31.520659 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd191dc-3ff4-4451-b93a-da68f606c84e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d\" (UID: \"afd191dc-3ff4-4451-b93a-da68f606c84e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:01:31.520901 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:31.520729 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd191dc-3ff4-4451-b93a-da68f606c84e-metrics-certs\") pod \"maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d\" (UID: \"afd191dc-3ff4-4451-b93a-da68f606c84e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:01:31.520901 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:01:31.520805 4947 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-serving-cert: secret "maas-api-e2e-extmod-a-1f83c0-serving-cert" not found Nov 03 17:01:31.520901 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:31.520838 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd191dc-3ff4-4451-b93a-da68f606c84e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d\" (UID: \"afd191dc-3ff4-4451-b93a-da68f606c84e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:01:31.520901 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:01:31.520875 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/afd191dc-3ff4-4451-b93a-da68f606c84e-maas-api-tls podName:afd191dc-3ff4-4451-b93a-da68f606c84e nodeName:}" failed. No retries permitted until 2026-11-03 17:01:32.020859308 +0000 UTC m=+2835.609859254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/afd191dc-3ff4-4451-b93a-da68f606c84e-maas-api-tls") pod "maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" (UID: "afd191dc-3ff4-4451-b93a-da68f606c84e") : secret "maas-api-e2e-extmod-a-1f83c0-serving-cert" not found Nov 03 17:01:31.521496 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:31.521477 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd191dc-3ff4-4451-b93a-da68f606c84e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d\" (UID: \"afd191dc-3ff4-4451-b93a-da68f606c84e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:01:31.523406 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:31.523383 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd191dc-3ff4-4451-b93a-da68f606c84e-metrics-certs\") pod \"maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d\" (UID: \"afd191dc-3ff4-4451-b93a-da68f606c84e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:01:31.560732 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:31.557520 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9x8m\" (UniqueName: \"kubernetes.io/projected/afd191dc-3ff4-4451-b93a-da68f606c84e-kube-api-access-g9x8m\") pod \"maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d\" (UID: \"afd191dc-3ff4-4451-b93a-da68f606c84e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:01:32.027453 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:32.027413 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd191dc-3ff4-4451-b93a-da68f606c84e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d\" (UID: \"afd191dc-3ff4-4451-b93a-da68f606c84e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:01:32.030090 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:32.030065 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd191dc-3ff4-4451-b93a-da68f606c84e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d\" (UID: \"afd191dc-3ff4-4451-b93a-da68f606c84e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:01:32.265442 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:32.265399 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:01:32.596677 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:32.596646 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d"] Nov 03 17:01:32.598629 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:01:32.598602 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd191dc_3ff4_4451_b93a_da68f606c84e.slice/crio-86b82a57ee851ea308f4f340c32ace4bded030dea9717c10b96b3bc4a9968774 WatchSource:0}: Error finding container 86b82a57ee851ea308f4f340c32ace4bded030dea9717c10b96b3bc4a9968774: Status 404 returned error can't find the container with id 86b82a57ee851ea308f4f340c32ace4bded030dea9717c10b96b3bc4a9968774 Nov 03 17:01:33.099002 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:33.098965 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" event={"ID":"afd191dc-3ff4-4451-b93a-da68f606c84e","Type":"ContainerStarted","Data":"be183b1d7c29d9451bac33d2f99dbb254aac4226e86122290c7d93eb848a6320"} Nov 03 17:01:33.099172 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:33.099018 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" event={"ID":"afd191dc-3ff4-4451-b93a-da68f606c84e","Type":"ContainerStarted","Data":"86b82a57ee851ea308f4f340c32ace4bded030dea9717c10b96b3bc4a9968774"} Nov 03 17:01:33.099221 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:33.099166 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:01:33.149956 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:33.149892 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" podStartSLOduration=1.828729286 podStartE2EDuration="2.149878128s" totalImagesPullingTime="321.148842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:33.146090626 +0000 UTC m=+2836.735090593" watchObservedRunningTime="2026-11-03 17:01:33.149878128 +0000 UTC m=+2836.738878095" Nov 03 17:01:35.043215 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:35.043183 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" Nov 03 17:01:39.112805 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:39.112774 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:01:41.395240 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.395206 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz"] Nov 03 17:01:41.401793 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.401770 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" Nov 03 17:01:41.421264 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.421236 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-3a0e27\"" Nov 03 17:01:41.440890 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.440859 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz"] Nov 03 17:01:41.445795 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.445763 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx"] Nov 03 17:01:41.451451 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.451427 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" Nov 03 17:01:41.465365 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.465332 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx"] Nov 03 17:01:41.467507 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.467474 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-3a0e27-dockercfg-n4qf6\"" Nov 03 17:01:41.497532 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.497477 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tknbh\" (UniqueName: \"kubernetes.io/projected/7bf7a98e-51f6-4ad2-b400-4715880c82d7-kube-api-access-tknbh\") pod \"payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz\" (UID: \"7bf7a98e-51f6-4ad2-b400-4715880c82d7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" Nov 03 17:01:41.497532 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.497531 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf7a98e-51f6-4ad2-b400-4715880c82d7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz\" (UID: \"7bf7a98e-51f6-4ad2-b400-4715880c82d7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" Nov 03 17:01:41.497857 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.497635 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpnbp\" (UniqueName: \"kubernetes.io/projected/786125e1-80ab-40fc-992d-57ddef757ddd-kube-api-access-tpnbp\") pod \"payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx\" (UID: \"786125e1-80ab-40fc-992d-57ddef757ddd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" Nov 03 17:01:41.497857 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.497828 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786125e1-80ab-40fc-992d-57ddef757ddd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx\" (UID: \"786125e1-80ab-40fc-992d-57ddef757ddd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" Nov 03 17:01:41.638724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.637964 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tknbh\" (UniqueName: \"kubernetes.io/projected/7bf7a98e-51f6-4ad2-b400-4715880c82d7-kube-api-access-tknbh\") pod \"payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz\" (UID: \"7bf7a98e-51f6-4ad2-b400-4715880c82d7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" Nov 03 17:01:41.638724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.638035 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf7a98e-51f6-4ad2-b400-4715880c82d7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz\" (UID: \"7bf7a98e-51f6-4ad2-b400-4715880c82d7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" Nov 03 17:01:41.638724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.638137 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpnbp\" (UniqueName: \"kubernetes.io/projected/786125e1-80ab-40fc-992d-57ddef757ddd-kube-api-access-tpnbp\") pod \"payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx\" (UID: \"786125e1-80ab-40fc-992d-57ddef757ddd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" Nov 03 17:01:41.638724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.638362 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786125e1-80ab-40fc-992d-57ddef757ddd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx\" (UID: \"786125e1-80ab-40fc-992d-57ddef757ddd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" Nov 03 17:01:41.649767 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.647889 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf7a98e-51f6-4ad2-b400-4715880c82d7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz\" (UID: \"7bf7a98e-51f6-4ad2-b400-4715880c82d7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" Nov 03 17:01:41.702850 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.700621 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786125e1-80ab-40fc-992d-57ddef757ddd-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx\" (UID: \"786125e1-80ab-40fc-992d-57ddef757ddd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" Nov 03 17:01:41.738437 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.738399 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tknbh\" (UniqueName: \"kubernetes.io/projected/7bf7a98e-51f6-4ad2-b400-4715880c82d7-kube-api-access-tknbh\") pod \"payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz\" (UID: \"7bf7a98e-51f6-4ad2-b400-4715880c82d7\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" Nov 03 17:01:41.743220 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.741603 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpnbp\" (UniqueName: \"kubernetes.io/projected/786125e1-80ab-40fc-992d-57ddef757ddd-kube-api-access-tpnbp\") pod \"payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx\" (UID: \"786125e1-80ab-40fc-992d-57ddef757ddd\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" Nov 03 17:01:41.791723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:41.778181 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" Nov 03 17:01:42.015976 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:42.015942 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" Nov 03 17:01:42.118685 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:42.118651 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx"] Nov 03 17:01:42.165673 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:42.165647 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" event={"ID":"786125e1-80ab-40fc-992d-57ddef757ddd","Type":"ContainerStarted","Data":"4e6f9604594fd7e2959383a3d359fa5d2572a7f2f32fe80eaf4271934f539b18"} Nov 03 17:01:42.344153 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:42.344127 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz"] Nov 03 17:01:42.344153 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:01:42.344138 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf7a98e_51f6_4ad2_b400_4715880c82d7.slice/crio-1fdc37be9d72d7651ab99aacc4ff7db2f40e908a23cca6f678df4eb98af44f42 WatchSource:0}: Error finding container 1fdc37be9d72d7651ab99aacc4ff7db2f40e908a23cca6f678df4eb98af44f42: Status 404 returned error can't find the container with id 1fdc37be9d72d7651ab99aacc4ff7db2f40e908a23cca6f678df4eb98af44f42 Nov 03 17:01:43.178491 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:43.178202 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" event={"ID":"786125e1-80ab-40fc-992d-57ddef757ddd","Type":"ContainerStarted","Data":"47a720a2568e251dc2e878bdf0f4bc326bbd2cd33d75785d5dc6a5076dd0f14d"} Nov 03 17:01:43.178491 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:43.178474 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" Nov 03 17:01:43.180533 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:43.180509 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" event={"ID":"7bf7a98e-51f6-4ad2-b400-4715880c82d7","Type":"ContainerStarted","Data":"e010114e224003eff5c4846ff124b837287118082aa5aa7710d03fe7a01d92da"} Nov 03 17:01:43.180688 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:43.180561 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" Nov 03 17:01:43.180688 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:43.180586 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" event={"ID":"7bf7a98e-51f6-4ad2-b400-4715880c82d7","Type":"ContainerStarted","Data":"1fdc37be9d72d7651ab99aacc4ff7db2f40e908a23cca6f678df4eb98af44f42"} Nov 03 17:01:43.296274 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:43.296225 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" podStartSLOduration=2.296209937 podStartE2EDuration="2.296209937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:43.244883818 +0000 UTC m=+2846.833883791" watchObservedRunningTime="2026-11-03 17:01:43.296209937 +0000 UTC m=+2846.885209949" Nov 03 17:01:43.298402 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:43.298365 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" podStartSLOduration=2.298352141 podStartE2EDuration="2.298352141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:43.296077133 +0000 UTC m=+2846.885077101" watchObservedRunningTime="2026-11-03 17:01:43.298352141 +0000 UTC m=+2846.887352110" Nov 03 17:01:47.878721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:47.874376 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:01:52.179423 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.179389 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c"] Nov 03 17:01:52.184055 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.184034 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.224223 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.224190 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c"] Nov 03 17:01:52.263025 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.262893 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-workload-certs\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.263823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.263059 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-workload-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.263823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.263104 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05078fe5-de88-4050-829f-d2a51f0328a8-istio-podinfo\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.263823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.263187 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-istio-data\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.263823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.263220 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-istio-envoy\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.263823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.263246 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05078fe5-de88-4050-829f-d2a51f0328a8-istiod-ca-cert\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.263823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.263296 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs2pt\" (UniqueName: \"kubernetes.io/projected/05078fe5-de88-4050-829f-d2a51f0328a8-kube-api-access-cs2pt\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.263823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.263430 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-credential-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.263823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.263482 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05078fe5-de88-4050-829f-d2a51f0328a8-istio-token\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.381825 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.381787 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-workload-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.383249 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.383225 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05078fe5-de88-4050-829f-d2a51f0328a8-istio-podinfo\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.392739 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.391947 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-istio-data\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.392739 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.391995 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-istio-envoy\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.392739 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.392028 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05078fe5-de88-4050-829f-d2a51f0328a8-istiod-ca-cert\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.392739 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.392088 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs2pt\" (UniqueName: \"kubernetes.io/projected/05078fe5-de88-4050-829f-d2a51f0328a8-kube-api-access-cs2pt\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.392739 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.392191 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-credential-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.392739 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.392246 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05078fe5-de88-4050-829f-d2a51f0328a8-istio-token\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.392739 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.392305 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-workload-certs\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.393221 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.392842 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-workload-certs\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.393221 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.383140 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-workload-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.393221 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.393144 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-istio-data\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.405718 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.398016 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-credential-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.406423 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.406394 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05078fe5-de88-4050-829f-d2a51f0328a8-istiod-ca-cert\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.434762 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.434425 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-istio-envoy\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.437309 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.437180 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05078fe5-de88-4050-829f-d2a51f0328a8-istio-token\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.438031 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.438004 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05078fe5-de88-4050-829f-d2a51f0328a8-istio-podinfo\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.440736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.438794 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs2pt\" (UniqueName: \"kubernetes.io/projected/05078fe5-de88-4050-829f-d2a51f0328a8-kube-api-access-cs2pt\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.503743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.501677 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:52.836265 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:01:52.836232 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05078fe5_de88_4050_829f_d2a51f0328a8.slice/crio-204e9b42e466480ce412f3f04ae913678b0c77030a67bb4d3f755d1147b7406b WatchSource:0}: Error finding container 204e9b42e466480ce412f3f04ae913678b0c77030a67bb4d3f755d1147b7406b: Status 404 returned error can't find the container with id 204e9b42e466480ce412f3f04ae913678b0c77030a67bb4d3f755d1147b7406b Nov 03 17:01:52.837321 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.837297 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c"] Nov 03 17:01:52.839326 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.839292 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:01:52.839423 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.839362 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:01:52.839423 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:52.839399 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:01:53.266746 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:53.266687 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" event={"ID":"05078fe5-de88-4050-829f-d2a51f0328a8","Type":"ContainerStarted","Data":"25eee918399d70e921a6a8551df2979b8c872cc32eadea09dadca047825cecc7"} Nov 03 17:01:53.267256 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:53.266778 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" event={"ID":"05078fe5-de88-4050-829f-d2a51f0328a8","Type":"ContainerStarted","Data":"204e9b42e466480ce412f3f04ae913678b0c77030a67bb4d3f755d1147b7406b"} Nov 03 17:01:53.315653 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:53.315594 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" podStartSLOduration=1.315576237 podStartE2EDuration="1.315576237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:01:53.314474093 +0000 UTC m=+2856.903474060" watchObservedRunningTime="2026-11-03 17:01:53.315576237 +0000 UTC m=+2856.904576206" Nov 03 17:01:53.503749 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:53.503686 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:01:53.505308 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:53.505280 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 17:01:53.505447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:53.505328 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" podUID="05078fe5-de88-4050-829f-d2a51f0328a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 17:01:54.193171 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:54.193143 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" Nov 03 17:01:54.193352 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:54.193213 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" Nov 03 17:01:54.511718 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:54.504199 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 17:01:54.511718 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:54.504272 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" podUID="05078fe5-de88-4050-829f-d2a51f0328a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 17:01:55.511724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:55.505072 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 17:01:55.511724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:55.505134 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" podUID="05078fe5-de88-4050-829f-d2a51f0328a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 17:01:56.520729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:56.520252 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 17:01:56.520729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:56.520307 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" podUID="05078fe5-de88-4050-829f-d2a51f0328a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 17:01:57.508729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:57.504411 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 17:01:57.508729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:57.504474 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" podUID="05078fe5-de88-4050-829f-d2a51f0328a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 17:01:58.525679 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:58.522112 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 17:01:58.525679 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:58.522186 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" podUID="05078fe5-de88-4050-829f-d2a51f0328a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 17:01:59.519722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:59.519088 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Nov 03 17:01:59.519722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:01:59.519147 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" podUID="05078fe5-de88-4050-829f-d2a51f0328a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Nov 03 17:02:00.132816 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:02:00.132780 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:02:00.508121 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:00.508092 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:02:00.508579 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:00.508564 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:02:00.509354 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:00.509332 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:02:15.597813 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.597760 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk"] Nov 03 17:02:15.602635 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.602600 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.628208 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.628181 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk"] Nov 03 17:02:15.654502 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.654471 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-workload-certs\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.654502 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.654507 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfpmb\" (UniqueName: \"kubernetes.io/projected/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-kube-api-access-zfpmb\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.654814 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.654665 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-data\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.654814 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.654727 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-credential-socket\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.654814 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.654811 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-workload-socket\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.655021 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.654832 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-podinfo\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.655021 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.654991 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-envoy\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.655172 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.655147 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-token\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.655249 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.655232 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istiod-ca-cert\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.683623 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.683599 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-fdbf0c-openshift-default-dockercfg-wq772\"" Nov 03 17:02:15.762415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.761480 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-token\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.762415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.761551 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istiod-ca-cert\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.762415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.761636 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-workload-certs\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.762415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.761665 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfpmb\" (UniqueName: \"kubernetes.io/projected/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-kube-api-access-zfpmb\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.762415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.761744 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-data\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.762415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.761773 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-credential-socket\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.762415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.761812 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-workload-socket\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.762415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.761835 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-podinfo\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.762415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.761915 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-envoy\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.774720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.767904 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istiod-ca-cert\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.774720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.768129 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-workload-certs\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.774720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.768378 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-credential-socket\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.775023 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.774965 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-envoy\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.784728 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.775229 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-workload-socket\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.786793 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.786767 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-data\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.817962 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.810458 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-podinfo\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.818973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.818653 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-token\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.819818 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.819750 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfpmb\" (UniqueName: \"kubernetes.io/projected/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-kube-api-access-zfpmb\") pod \"e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:15.927771 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:15.927015 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:16.305327 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:02:16.304081 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba72dc1_f7d9_4cb5_ae6a_3c20337f3d3e.slice/crio-c9b194aa96cb065a2e2409f2ee068f5a32af3957d7fb09bd367694afc64f100a WatchSource:0}: Error finding container c9b194aa96cb065a2e2409f2ee068f5a32af3957d7fb09bd367694afc64f100a: Status 404 returned error can't find the container with id c9b194aa96cb065a2e2409f2ee068f5a32af3957d7fb09bd367694afc64f100a Nov 03 17:02:16.305327 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:16.304921 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk"] Nov 03 17:02:16.306865 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:16.306831 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:02:16.306986 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:16.306906 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:02:16.306986 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:16.306947 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:02:16.509585 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:16.509550 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" event={"ID":"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e","Type":"ContainerStarted","Data":"ddc9ab561808b41b10a6fe5eff8401fde77078b66c59bb58fd9579b47172a5f7"} Nov 03 17:02:16.509795 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:16.509609 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" event={"ID":"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e","Type":"ContainerStarted","Data":"c9b194aa96cb065a2e2409f2ee068f5a32af3957d7fb09bd367694afc64f100a"} Nov 03 17:02:16.561639 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:16.561539 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" podStartSLOduration=1.561525737 podStartE2EDuration="1.561525737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:16.557561117 +0000 UTC m=+2880.146561084" watchObservedRunningTime="2026-11-03 17:02:16.561525737 +0000 UTC m=+2880.150525704" Nov 03 17:02:16.927990 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:16.927912 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:17.516209 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:17.506347 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:02:17.935158 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:17.935081 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:18.654247 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:18.654215 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:18.654462 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:18.654313 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:02:23.455779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.455742 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr"] Nov 03 17:02:23.461579 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.461549 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.488981 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.488945 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr"] Nov 03 17:02:23.551387 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.551358 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.551567 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.551405 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.551610 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.551580 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.551646 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.551614 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.551771 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.551755 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.551815 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.551787 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f0e126-f94a-4cb6-b187-e05f23476696-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.551849 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.551823 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.551942 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.551924 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7fgv\" (UniqueName: \"kubernetes.io/projected/d9f0e126-f94a-4cb6-b187-e05f23476696-kube-api-access-g7fgv\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.551981 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.551967 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.559773 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.559750 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-1f83c0-openshift-default-dockercfg-bw8dj\"" Nov 03 17:02:23.653943 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.653899 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.654205 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.653959 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.654205 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.654118 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.654205 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.654184 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f0e126-f94a-4cb6-b187-e05f23476696-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.654400 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.654241 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.654400 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.654338 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7fgv\" (UniqueName: \"kubernetes.io/projected/d9f0e126-f94a-4cb6-b187-e05f23476696-kube-api-access-g7fgv\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.654400 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.654389 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.654566 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.654490 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.654566 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.654540 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.656121 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.655894 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.657887 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.657853 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.659254 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.659192 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.662217 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.662192 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.662633 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.662608 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.668210 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.668177 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.670129 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.670105 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f0e126-f94a-4cb6-b187-e05f23476696-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.682381 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.682356 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.682572 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.682553 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7fgv\" (UniqueName: \"kubernetes.io/projected/d9f0e126-f94a-4cb6-b187-e05f23476696-kube-api-access-g7fgv\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:23.779054 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:23.779015 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:24.109513 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:02:24.109478 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f0e126_f94a_4cb6_b187_e05f23476696.slice/crio-c3f965f77ed836e360dab3f466d9639484acdd7df2d2e37d66886c82c4adb838 WatchSource:0}: Error finding container c3f965f77ed836e360dab3f466d9639484acdd7df2d2e37d66886c82c4adb838: Status 404 returned error can't find the container with id c3f965f77ed836e360dab3f466d9639484acdd7df2d2e37d66886c82c4adb838 Nov 03 17:02:24.110778 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:24.110751 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr"] Nov 03 17:02:24.112670 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:24.112638 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:02:24.112781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:24.112765 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:02:24.112822 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:24.112802 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:02:24.645198 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:24.645163 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" event={"ID":"d9f0e126-f94a-4cb6-b187-e05f23476696","Type":"ContainerStarted","Data":"3dff21d654da0f2332a9f2530d080428f0be198405c87d22cfb457c427e2b694"} Nov 03 17:02:24.645639 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:24.645222 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" event={"ID":"d9f0e126-f94a-4cb6-b187-e05f23476696","Type":"ContainerStarted","Data":"c3f965f77ed836e360dab3f466d9639484acdd7df2d2e37d66886c82c4adb838"} Nov 03 17:02:24.719252 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:24.719181 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" podStartSLOduration=1.719158309 podStartE2EDuration="1.719158309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:24.696505447 +0000 UTC m=+2888.285505413" watchObservedRunningTime="2026-11-03 17:02:24.719158309 +0000 UTC m=+2888.308158277" Nov 03 17:02:24.779677 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:24.779642 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:25.781618 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:25.781584 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 17:02:25.782087 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:25.781649 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" podUID="d9f0e126-f94a-4cb6-b187-e05f23476696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 17:02:26.779597 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:26.779518 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 17:02:26.779812 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:26.779618 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" podUID="d9f0e126-f94a-4cb6-b187-e05f23476696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 17:02:27.779615 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:27.779581 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 17:02:27.780145 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:27.779638 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" podUID="d9f0e126-f94a-4cb6-b187-e05f23476696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 17:02:28.792418 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:28.792384 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 17:02:28.793432 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:28.793394 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" podUID="d9f0e126-f94a-4cb6-b187-e05f23476696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 17:02:29.779610 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:29.779573 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 17:02:29.779897 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:29.779633 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" podUID="d9f0e126-f94a-4cb6-b187-e05f23476696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 17:02:30.133106 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:02:30.133016 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:02:30.784907 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:30.784875 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:30.785323 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:30.785302 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:30.786137 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:30.786118 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:02:47.216888 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.216839 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw"] Nov 03 17:02:47.221631 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.221612 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" Nov 03 17:02:47.243066 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.243031 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw"] Nov 03 17:02:47.273869 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.253962 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"payload-processing-external-model-e2e--40ad01b7-dockercfg-5l5bn\"" Nov 03 17:02:47.273869 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.254009 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"agc-cred-6d6e4840c5a9\"" Nov 03 17:02:47.273869 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.256411 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6046c2\"" Nov 03 17:02:47.273869 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.256500 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"kube-root-ca.crt\"" Nov 03 17:02:47.273869 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.256538 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"openshift-service-ca.crt\"" Nov 03 17:02:47.273869 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.256600 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"routing-overlay\"" Nov 03 17:02:47.273869 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.263957 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61811daa-ebc2-41b4-a703-dfbf66f54ed2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw\" (UID: \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" Nov 03 17:02:47.273869 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.264076 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61811daa-ebc2-41b4-a703-dfbf66f54ed2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw\" (UID: \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" Nov 03 17:02:47.273869 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.264207 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9zt9\" (UniqueName: \"kubernetes.io/projected/61811daa-ebc2-41b4-a703-dfbf66f54ed2-kube-api-access-f9zt9\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw\" (UID: \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" Nov 03 17:02:47.273869 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.264327 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61811daa-ebc2-41b4-a703-dfbf66f54ed2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw\" (UID: \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" Nov 03 17:02:47.366328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.366255 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9zt9\" (UniqueName: \"kubernetes.io/projected/61811daa-ebc2-41b4-a703-dfbf66f54ed2-kube-api-access-f9zt9\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw\" (UID: \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" Nov 03 17:02:47.367355 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.367326 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61811daa-ebc2-41b4-a703-dfbf66f54ed2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw\" (UID: \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" Nov 03 17:02:47.414720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.400079 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61811daa-ebc2-41b4-a703-dfbf66f54ed2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw\" (UID: \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" Nov 03 17:02:47.416425 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.396442 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61811daa-ebc2-41b4-a703-dfbf66f54ed2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw\" (UID: \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" Nov 03 17:02:47.416818 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.416680 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61811daa-ebc2-41b4-a703-dfbf66f54ed2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw\" (UID: \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" Nov 03 17:02:47.418829 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.418326 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61811daa-ebc2-41b4-a703-dfbf66f54ed2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw\" (UID: \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" Nov 03 17:02:47.443589 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.439861 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9zt9\" (UniqueName: \"kubernetes.io/projected/61811daa-ebc2-41b4-a703-dfbf66f54ed2-kube-api-access-f9zt9\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw\" (UID: \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" Nov 03 17:02:47.443855 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.443476 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61811daa-ebc2-41b4-a703-dfbf66f54ed2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw\" (UID: \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" Nov 03 17:02:47.537435 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.537086 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" Nov 03 17:02:47.867357 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:02:47.867327 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61811daa_ebc2_41b4_a703_dfbf66f54ed2.slice/crio-c9702011e02ba0bc40e842e5265caa944c654f7bd3d9106323ab07c067efc003 WatchSource:0}: Error finding container c9702011e02ba0bc40e842e5265caa944c654f7bd3d9106323ab07c067efc003: Status 404 returned error can't find the container with id c9702011e02ba0bc40e842e5265caa944c654f7bd3d9106323ab07c067efc003 Nov 03 17:02:47.867823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:47.867803 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw"] Nov 03 17:02:48.878908 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:48.878873 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" event={"ID":"61811daa-ebc2-41b4-a703-dfbf66f54ed2","Type":"ContainerStarted","Data":"f677f506f04ed33b1211e8bd9972ab6a67e8f0e2e244013af592ea5d9b3ec1ef"} Nov 03 17:02:48.879319 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:48.878925 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" event={"ID":"61811daa-ebc2-41b4-a703-dfbf66f54ed2","Type":"ContainerStarted","Data":"c9702011e02ba0bc40e842e5265caa944c654f7bd3d9106323ab07c067efc003"} Nov 03 17:02:48.879319 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:48.879026 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" Nov 03 17:02:48.927840 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:48.927796 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" podStartSLOduration=1.927782962 podStartE2EDuration="1.927782962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:48.925968525 +0000 UTC m=+2912.514968493" watchObservedRunningTime="2026-11-03 17:02:48.927782962 +0000 UTC m=+2912.516782930" Nov 03 17:02:53.543777 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.543737 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj"] Nov 03 17:02:53.550121 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.550095 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.581368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.581334 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj"] Nov 03 17:02:53.598633 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.598606 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v"] Nov 03 17:02:53.603149 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.603128 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.621607 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.619837 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v"] Nov 03 17:02:53.624660 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.624626 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.625386 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.625361 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.625542 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.625521 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.625741 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.625721 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.625948 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.625925 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.626101 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.626083 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.626232 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.626216 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7425c4a-724e-4835-b34d-47b69913caf2-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.626359 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.626344 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.626481 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.626465 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.626612 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.626592 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.626855 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.626834 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7425c4a-724e-4835-b34d-47b69913caf2-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.627089 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.627039 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw59s\" (UniqueName: \"kubernetes.io/projected/732aa07c-3673-424e-a9a8-e7c2a5d5520a-kube-api-access-pw59s\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.627191 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.627154 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.627260 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.627206 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.627322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.627262 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.627322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.627296 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7425c4a-724e-4835-b34d-47b69913caf2-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.627438 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.627340 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.627438 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.627391 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z86j\" (UniqueName: \"kubernetes.io/projected/d7425c4a-724e-4835-b34d-47b69913caf2-kube-api-access-7z86j\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.729291 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.729246 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7425c4a-724e-4835-b34d-47b69913caf2-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.729492 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.729309 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw59s\" (UniqueName: \"kubernetes.io/projected/732aa07c-3673-424e-a9a8-e7c2a5d5520a-kube-api-access-pw59s\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.729492 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.729380 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.729492 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.729422 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.729492 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.729471 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.729791 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.729499 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7425c4a-724e-4835-b34d-47b69913caf2-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.729791 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.729539 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.729791 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.729582 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z86j\" (UniqueName: \"kubernetes.io/projected/d7425c4a-724e-4835-b34d-47b69913caf2-kube-api-access-7z86j\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.729791 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.729680 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.729791 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.729780 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.730067 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.729815 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.730067 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.729854 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.730067 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.729896 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.730067 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.729903 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.730067 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.729936 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.730067 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.729954 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.730067 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.729972 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7425c4a-724e-4835-b34d-47b69913caf2-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.730067 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.730006 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.730067 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.730038 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.730618 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.730073 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.730618 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.730257 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.730618 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.730346 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.730618 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.730426 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-workload-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.730618 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.730536 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.730903 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.730740 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-workload-certs\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.730903 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.730828 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-credential-socket\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.731224 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.731176 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-istio-data\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.731563 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.731538 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7425c4a-724e-4835-b34d-47b69913caf2-istiod-ca-cert\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.732579 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.732552 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7425c4a-724e-4835-b34d-47b69913caf2-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.732854 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.732837 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-podinfo\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.733173 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.733156 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.733822 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.733805 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-envoy\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.754163 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.754070 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7425c4a-724e-4835-b34d-47b69913caf2-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.754316 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.754241 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z86j\" (UniqueName: \"kubernetes.io/projected/d7425c4a-724e-4835-b34d-47b69913caf2-kube-api-access-7z86j\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.754408 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.754385 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw59s\" (UniqueName: \"kubernetes.io/projected/732aa07c-3673-424e-a9a8-e7c2a5d5520a-kube-api-access-pw59s\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.755799 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.755776 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-token\") pod \"e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:53.866132 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.866044 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:53.926224 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:53.926177 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:54.193532 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:02:54.193504 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7425c4a_724e_4835_b34d_47b69913caf2.slice/crio-d97b91c417943c5350891dade707c544f3b7c7298d6bb3e3f21f175a49817273 WatchSource:0}: Error finding container d97b91c417943c5350891dade707c544f3b7c7298d6bb3e3f21f175a49817273: Status 404 returned error can't find the container with id d97b91c417943c5350891dade707c544f3b7c7298d6bb3e3f21f175a49817273 Nov 03 17:02:54.196634 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:54.196593 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:02:54.196765 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:54.196673 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:02:54.196765 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:54.196722 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj"] Nov 03 17:02:54.196897 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:54.196732 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:02:54.267100 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:54.267064 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v"] Nov 03 17:02:54.271086 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:54.271043 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:02:54.271293 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:54.271138 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:02:54.271293 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:54.271183 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:02:54.927500 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:54.927457 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" event={"ID":"d7425c4a-724e-4835-b34d-47b69913caf2","Type":"ContainerStarted","Data":"1fab539209cbd79c1513a14ed573f5e8cfffd2e22de651805ddeec112b193d59"} Nov 03 17:02:54.928013 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:54.927534 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" event={"ID":"d7425c4a-724e-4835-b34d-47b69913caf2","Type":"ContainerStarted","Data":"d97b91c417943c5350891dade707c544f3b7c7298d6bb3e3f21f175a49817273"} Nov 03 17:02:54.929265 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:54.929236 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" event={"ID":"732aa07c-3673-424e-a9a8-e7c2a5d5520a","Type":"ContainerStarted","Data":"88d1fe2bc1830f3f482b48194c727d28effabaaf6804cfd4a847ffb12deff4bd"} Nov 03 17:02:54.929386 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:54.929286 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" event={"ID":"732aa07c-3673-424e-a9a8-e7c2a5d5520a","Type":"ContainerStarted","Data":"9d38aa8a2618dc109cb207acbc578f109ab6832ff89d39c49980cfca19850857"} Nov 03 17:02:55.009327 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:55.009271 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" podStartSLOduration=2.009256871 podStartE2EDuration="2.009256871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:54.993226833 +0000 UTC m=+2918.582226802" watchObservedRunningTime="2026-11-03 17:02:55.009256871 +0000 UTC m=+2918.598256839" Nov 03 17:02:55.047399 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:55.047339 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" podStartSLOduration=2.047321198 podStartE2EDuration="2.047321198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:02:55.044421458 +0000 UTC m=+2918.633421427" watchObservedRunningTime="2026-11-03 17:02:55.047321198 +0000 UTC m=+2918.636321169" Nov 03 17:02:55.866931 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:55.866900 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:02:55.868578 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:55.868554 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 17:02:55.868773 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:55.868597 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" podUID="d7425c4a-724e-4835-b34d-47b69913caf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 17:02:55.937151 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:55.937111 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:02:55.952727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:55.950923 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 17:02:55.952727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:55.950993 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" podUID="732aa07c-3673-424e-a9a8-e7c2a5d5520a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 17:02:56.867262 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:56.867225 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 17:02:56.867444 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:56.867276 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" podUID="d7425c4a-724e-4835-b34d-47b69913caf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 17:02:56.928732 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:56.928385 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 17:02:56.928732 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:56.928445 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" podUID="732aa07c-3673-424e-a9a8-e7c2a5d5520a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 17:02:57.867117 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:57.867074 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 17:02:57.867524 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:57.867153 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" podUID="d7425c4a-724e-4835-b34d-47b69913caf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 17:02:57.928729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:57.928124 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 17:02:57.928729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:57.928207 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" podUID="732aa07c-3673-424e-a9a8-e7c2a5d5520a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 17:02:58.868756 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:58.867338 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 17:02:58.868756 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:58.867434 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" podUID="d7425c4a-724e-4835-b34d-47b69913caf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 17:02:58.944727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:58.944161 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 17:02:58.944727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:58.944245 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" podUID="732aa07c-3673-424e-a9a8-e7c2a5d5520a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 17:02:59.868241 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:59.868203 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 17:02:59.868425 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:59.868259 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" podUID="d7425c4a-724e-4835-b34d-47b69913caf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 17:02:59.889263 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:59.889225 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" Nov 03 17:02:59.935726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:59.935209 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 17:02:59.935726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:02:59.935266 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" podUID="732aa07c-3673-424e-a9a8-e7c2a5d5520a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 17:03:00.390015 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:03:00.389978 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:03:00.870262 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:00.870217 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Nov 03 17:03:00.870574 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:00.870299 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" podUID="d7425c4a-724e-4835-b34d-47b69913caf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Nov 03 17:03:00.937724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:00.937126 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 17:03:00.937724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:00.937190 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" podUID="732aa07c-3673-424e-a9a8-e7c2a5d5520a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 17:03:01.872434 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:01.872405 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:03:01.872835 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:01.872813 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:03:01.873842 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:01.873824 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:03:01.932424 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:01.932390 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:03:01.933102 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:01.933073 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:03:01.934019 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:01.933996 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:03:02.111448 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:02.111412 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:03:08.947282 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:08.947246 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf"] Nov 03 17:03:08.952056 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:08.952033 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" Nov 03 17:03:08.975314 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:08.975279 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf"] Nov 03 17:03:08.983908 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:08.983877 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-790111-dockercfg-57rvb\"" Nov 03 17:03:08.984317 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:08.984274 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-790111\"" Nov 03 17:03:09.117262 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:09.117220 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b5b746-c30a-428b-95ef-7f94cc3f6fd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf\" (UID: \"43b5b746-c30a-428b-95ef-7f94cc3f6fd7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" Nov 03 17:03:09.129722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:09.118043 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ddp6\" (UniqueName: \"kubernetes.io/projected/43b5b746-c30a-428b-95ef-7f94cc3f6fd7-kube-api-access-5ddp6\") pod \"payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf\" (UID: \"43b5b746-c30a-428b-95ef-7f94cc3f6fd7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" Nov 03 17:03:09.312920 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:09.312862 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b5b746-c30a-428b-95ef-7f94cc3f6fd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf\" (UID: \"43b5b746-c30a-428b-95ef-7f94cc3f6fd7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" Nov 03 17:03:09.313683 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:09.313651 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b5b746-c30a-428b-95ef-7f94cc3f6fd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf\" (UID: \"43b5b746-c30a-428b-95ef-7f94cc3f6fd7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" Nov 03 17:03:09.313974 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:09.313953 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ddp6\" (UniqueName: \"kubernetes.io/projected/43b5b746-c30a-428b-95ef-7f94cc3f6fd7-kube-api-access-5ddp6\") pod \"payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf\" (UID: \"43b5b746-c30a-428b-95ef-7f94cc3f6fd7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" Nov 03 17:03:09.383579 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:09.383534 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ddp6\" (UniqueName: \"kubernetes.io/projected/43b5b746-c30a-428b-95ef-7f94cc3f6fd7-kube-api-access-5ddp6\") pod \"payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf\" (UID: \"43b5b746-c30a-428b-95ef-7f94cc3f6fd7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" Nov 03 17:03:09.574219 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:09.574128 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" Nov 03 17:03:10.323550 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:03:10.323517 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b5b746_c30a_428b_95ef_7f94cc3f6fd7.slice/crio-7f462c5292436839251f629e292e085d8bc23c52cb5695c8cabf1dd191e95f79 WatchSource:0}: Error finding container 7f462c5292436839251f629e292e085d8bc23c52cb5695c8cabf1dd191e95f79: Status 404 returned error can't find the container with id 7f462c5292436839251f629e292e085d8bc23c52cb5695c8cabf1dd191e95f79 Nov 03 17:03:10.323935 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:10.323672 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf"] Nov 03 17:03:10.477373 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:10.477338 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" event={"ID":"43b5b746-c30a-428b-95ef-7f94cc3f6fd7","Type":"ContainerStarted","Data":"771f33136531bc0165308faaae1d3d5be0ae19b76651ae87a894a190e35d0f1f"} Nov 03 17:03:10.477584 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:10.477392 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" event={"ID":"43b5b746-c30a-428b-95ef-7f94cc3f6fd7","Type":"ContainerStarted","Data":"7f462c5292436839251f629e292e085d8bc23c52cb5695c8cabf1dd191e95f79"} Nov 03 17:03:10.477584 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:10.477492 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" Nov 03 17:03:10.525473 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:10.525420 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" podStartSLOduration=2.525405941 podStartE2EDuration="2.525405941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:03:10.521853483 +0000 UTC m=+2934.110853455" watchObservedRunningTime="2026-11-03 17:03:10.525405941 +0000 UTC m=+2934.114405909" Nov 03 17:03:11.158831 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.158761 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw"] Nov 03 17:03:11.164898 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.164865 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.165088 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.165019 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst"] Nov 03 17:03:11.169712 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.169671 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.182883 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.182852 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-workload-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.183056 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.182915 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c844d192-95a9-4170-8af6-1b421fb533b5-istio-token\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.183128 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.183088 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c844d192-95a9-4170-8af6-1b421fb533b5-istio-podinfo\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.183188 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.183167 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-credential-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.183252 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.183206 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c844d192-95a9-4170-8af6-1b421fb533b5-istiod-ca-cert\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.183310 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.183291 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-istio-data\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.183415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.183396 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-workload-certs\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.183569 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.183552 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nprwg\" (UniqueName: \"kubernetes.io/projected/c844d192-95a9-4170-8af6-1b421fb533b5-kube-api-access-nprwg\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.183640 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.183592 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-istio-envoy\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.187983 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.187956 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw"] Nov 03 17:03:11.207714 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.207655 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst"] Nov 03 17:03:11.229531 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.229499 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-ca9d7c-openshift-default-dockercfg-fgrm2\"" Nov 03 17:03:11.285145 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.285108 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-envoy\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.285348 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.285167 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nprwg\" (UniqueName: \"kubernetes.io/projected/c844d192-95a9-4170-8af6-1b421fb533b5-kube-api-access-nprwg\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.285348 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.285228 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-credential-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.285348 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.285326 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-istio-envoy\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.285470 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.285378 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-workload-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.285470 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.285421 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-token\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.285560 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.285466 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c844d192-95a9-4170-8af6-1b421fb533b5-istio-token\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.285805 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.285782 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ead42d5-f3f0-4e06-a578-d4d457639850-istiod-ca-cert\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.285905 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.285825 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-workload-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.285905 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.285833 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c844d192-95a9-4170-8af6-1b421fb533b5-istio-podinfo\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.285905 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.285858 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqsvv\" (UniqueName: \"kubernetes.io/projected/2ead42d5-f3f0-4e06-a578-d4d457639850-kube-api-access-gqsvv\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.286098 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.285963 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-credential-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.286098 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.286040 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c844d192-95a9-4170-8af6-1b421fb533b5-istiod-ca-cert\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.286098 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.286076 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-istio-data\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.286253 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.286152 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-workload-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.286253 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.286228 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-workload-certs\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.286353 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.286291 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-workload-certs\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.286404 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.286379 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-credential-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.286460 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.286426 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-data\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.286589 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.286521 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-podinfo\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.286651 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.286616 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-istio-data\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.286734 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.286718 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-workload-certs\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.287211 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.287185 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c844d192-95a9-4170-8af6-1b421fb533b5-istiod-ca-cert\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.288136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.288110 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-istio-envoy\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.289018 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.288997 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c844d192-95a9-4170-8af6-1b421fb533b5-istio-podinfo\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.368725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.368664 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nprwg\" (UniqueName: \"kubernetes.io/projected/c844d192-95a9-4170-8af6-1b421fb533b5-kube-api-access-nprwg\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.377722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.375217 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c844d192-95a9-4170-8af6-1b421fb533b5-istio-token\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.424978 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.424875 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ead42d5-f3f0-4e06-a578-d4d457639850-istiod-ca-cert\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.425259 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.425236 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqsvv\" (UniqueName: \"kubernetes.io/projected/2ead42d5-f3f0-4e06-a578-d4d457639850-kube-api-access-gqsvv\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.425476 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.425462 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-workload-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.425655 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.425641 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-workload-certs\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.425799 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.425782 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-data\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.425955 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.425932 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-podinfo\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.426153 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.426134 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-envoy\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.426338 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.426312 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-credential-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.426530 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.426515 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-token\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.428193 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.428167 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ead42d5-f3f0-4e06-a578-d4d457639850-istiod-ca-cert\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.445737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.429352 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-credential-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.445737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.429666 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-workload-certs\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.445737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.429954 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-data\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.445737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.441324 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-envoy\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.445737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.443467 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-podinfo\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.446427 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.446398 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-workload-socket\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.457737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.455890 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqsvv\" (UniqueName: \"kubernetes.io/projected/2ead42d5-f3f0-4e06-a578-d4d457639850-kube-api-access-gqsvv\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.469728 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.467376 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-token\") pod \"e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:11.496732 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.493213 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:11.511725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:11.502124 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:12.014722 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:03:12.014666 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc844d192_95a9_4170_8af6_1b421fb533b5.slice/crio-320c7470b4c2d8a598c8b779b1d22e28c8fc5758a9a5f0f20eb2f48bfc656ba4 WatchSource:0}: Error finding container 320c7470b4c2d8a598c8b779b1d22e28c8fc5758a9a5f0f20eb2f48bfc656ba4: Status 404 returned error can't find the container with id 320c7470b4c2d8a598c8b779b1d22e28c8fc5758a9a5f0f20eb2f48bfc656ba4 Nov 03 17:03:12.016052 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:12.016028 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw"] Nov 03 17:03:12.017965 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:12.017929 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:03:12.018058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:12.017996 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:03:12.018058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:12.018029 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:03:12.029069 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:12.029024 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst"] Nov 03 17:03:12.030386 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:03:12.030352 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ead42d5_f3f0_4e06_a578_d4d457639850.slice/crio-c481069bc25fdfbe5b5d2343884a7c0f0cc3e4523632d2e4e0d2a32835c2a887 WatchSource:0}: Error finding container c481069bc25fdfbe5b5d2343884a7c0f0cc3e4523632d2e4e0d2a32835c2a887: Status 404 returned error can't find the container with id c481069bc25fdfbe5b5d2343884a7c0f0cc3e4523632d2e4e0d2a32835c2a887 Nov 03 17:03:12.033589 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:12.033548 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:03:12.033743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:12.033629 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:03:12.033743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:12.033663 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:03:12.574310 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:12.574280 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" event={"ID":"2ead42d5-f3f0-4e06-a578-d4d457639850","Type":"ContainerStarted","Data":"12d8b7f2df598cb5e68a435196a97ee88f3cbb29a04d91adf99e0a515587cb66"} Nov 03 17:03:12.574763 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:12.574328 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" event={"ID":"2ead42d5-f3f0-4e06-a578-d4d457639850","Type":"ContainerStarted","Data":"c481069bc25fdfbe5b5d2343884a7c0f0cc3e4523632d2e4e0d2a32835c2a887"} Nov 03 17:03:12.576275 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:12.576241 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" event={"ID":"c844d192-95a9-4170-8af6-1b421fb533b5","Type":"ContainerStarted","Data":"b7096418373c3bdc542d7806ab775777fc04f1e1597b6e98aa7a110bd1ae14c0"} Nov 03 17:03:12.576423 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:12.576299 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" event={"ID":"c844d192-95a9-4170-8af6-1b421fb533b5","Type":"ContainerStarted","Data":"320c7470b4c2d8a598c8b779b1d22e28c8fc5758a9a5f0f20eb2f48bfc656ba4"} Nov 03 17:03:12.626797 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:12.626735 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" podStartSLOduration=1.6267175969999998 podStartE2EDuration="1.626717597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:03:12.623031134 +0000 UTC m=+2936.212031105" watchObservedRunningTime="2026-11-03 17:03:12.626717597 +0000 UTC m=+2936.215717570" Nov 03 17:03:12.673367 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:12.673302 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" podStartSLOduration=1.673281716 podStartE2EDuration="1.673281716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:03:12.671960216 +0000 UTC m=+2936.260960182" watchObservedRunningTime="2026-11-03 17:03:12.673281716 +0000 UTC m=+2936.262281686" Nov 03 17:03:13.506046 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:13.506000 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:13.506444 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:13.506414 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:13.535044 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:13.535006 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 17:03:13.535432 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:13.534967 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 17:03:13.535593 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:13.535573 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" podUID="2ead42d5-f3f0-4e06-a578-d4d457639850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 17:03:13.536096 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:13.535390 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" podUID="c844d192-95a9-4170-8af6-1b421fb533b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 17:03:14.505727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:14.505360 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 17:03:14.505727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:14.505296 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 17:03:14.506344 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:14.505472 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" podUID="c844d192-95a9-4170-8af6-1b421fb533b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 17:03:14.506344 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:14.505949 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" podUID="2ead42d5-f3f0-4e06-a578-d4d457639850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 17:03:15.511734 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:15.506408 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 17:03:15.511734 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:15.506473 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" podUID="2ead42d5-f3f0-4e06-a578-d4d457639850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 17:03:15.511734 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:15.506801 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 17:03:15.511734 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:15.506849 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" podUID="c844d192-95a9-4170-8af6-1b421fb533b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 17:03:16.529735 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:16.513826 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 17:03:16.529735 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:16.513900 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" podUID="c844d192-95a9-4170-8af6-1b421fb533b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 17:03:16.529735 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:16.514356 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 17:03:16.529735 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:16.514394 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" podUID="2ead42d5-f3f0-4e06-a578-d4d457639850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 17:03:17.529729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:17.524434 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 17:03:17.529729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:17.524517 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" podUID="2ead42d5-f3f0-4e06-a578-d4d457639850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 17:03:17.551727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:17.550900 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 17:03:17.551727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:17.551015 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" podUID="c844d192-95a9-4170-8af6-1b421fb533b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 17:03:18.516734 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:18.507774 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Nov 03 17:03:18.516734 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:18.507851 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" podUID="2ead42d5-f3f0-4e06-a578-d4d457639850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Nov 03 17:03:18.516734 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:18.508272 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body= Nov 03 17:03:18.516734 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:18.508313 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" podUID="c844d192-95a9-4170-8af6-1b421fb533b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" Nov 03 17:03:19.508279 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:19.508244 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:19.508686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:19.508325 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:19.508788 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:19.508748 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:19.508852 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:19.508823 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:19.509485 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:19.509464 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:03:19.509587 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:19.509513 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:03:21.533966 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:21.533920 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p"] Nov 03 17:03:21.534472 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:21.534198 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" podUID="adc797dd-c98b-4af6-8939-770401241bf0" containerName="istio-proxy" containerID="cri-o://c9bb15903a12cd75d6e3d2abd0c2ddd427a4bf156735c531f92a2dfa3335bc54" gracePeriod=30 Nov 03 17:03:21.564137 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:21.564104 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" Nov 03 17:03:26.772097 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.772041 4947 generic.go:356] "Generic (PLEG): container finished" podID="adc797dd-c98b-4af6-8939-770401241bf0" containerID="c9bb15903a12cd75d6e3d2abd0c2ddd427a4bf156735c531f92a2dfa3335bc54" exitCode=0 Nov 03 17:03:26.772097 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.772077 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" event={"ID":"adc797dd-c98b-4af6-8939-770401241bf0","Type":"ContainerDied","Data":"c9bb15903a12cd75d6e3d2abd0c2ddd427a4bf156735c531f92a2dfa3335bc54"} Nov 03 17:03:26.772747 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.772344 4947 scope.go:122] "RemoveContainer" containerID="bb75213e9c1538e13c87667b0d6e2801ce0f1d4c37630a6013cf489cd90caf0d" Nov 03 17:03:26.840120 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.840091 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 17:03:26.945057 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.945026 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adc797dd-c98b-4af6-8939-770401241bf0-kube-api-access-kc5gv\" (UniqueName: \"kubernetes.io/projected/adc797dd-c98b-4af6-8939-770401241bf0-kube-api-access-kc5gv\") pod \"adc797dd-c98b-4af6-8939-770401241bf0\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " Nov 03 17:03:26.945276 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.945078 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-credential-socket\") pod \"adc797dd-c98b-4af6-8939-770401241bf0\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " Nov 03 17:03:26.945276 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.945119 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adc797dd-c98b-4af6-8939-770401241bf0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adc797dd-c98b-4af6-8939-770401241bf0-istio-podinfo\") pod \"adc797dd-c98b-4af6-8939-770401241bf0\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " Nov 03 17:03:26.945276 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.945216 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-istio-envoy\") pod \"adc797dd-c98b-4af6-8939-770401241bf0\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " Nov 03 17:03:26.945477 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.945396 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adc797dd-c98b-4af6-8939-770401241bf0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adc797dd-c98b-4af6-8939-770401241bf0-istiod-ca-cert\") pod \"adc797dd-c98b-4af6-8939-770401241bf0\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " Nov 03 17:03:26.945477 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.945419 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-credential-socket" pod "adc797dd-c98b-4af6-8939-770401241bf0" (UID: "adc797dd-c98b-4af6-8939-770401241bf0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:03:26.945598 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.945576 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-workload-certs\") pod \"adc797dd-c98b-4af6-8939-770401241bf0\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " Nov 03 17:03:26.945654 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.945620 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-istio-data\") pod \"adc797dd-c98b-4af6-8939-770401241bf0\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " Nov 03 17:03:26.946110 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.945890 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adc797dd-c98b-4af6-8939-770401241bf0-istio-token\" (UniqueName: \"kubernetes.io/projected/adc797dd-c98b-4af6-8939-770401241bf0-istio-token\") pod \"adc797dd-c98b-4af6-8939-770401241bf0\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " Nov 03 17:03:26.946110 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.945899 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adc797dd-c98b-4af6-8939-770401241bf0-istiod-ca-cert" pod "adc797dd-c98b-4af6-8939-770401241bf0" (UID: "adc797dd-c98b-4af6-8939-770401241bf0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:03:26.946295 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.946073 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-workload-socket\") pod \"adc797dd-c98b-4af6-8939-770401241bf0\" (UID: \"adc797dd-c98b-4af6-8939-770401241bf0\") " Nov 03 17:03:26.946295 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.946218 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-istio-data" pod "adc797dd-c98b-4af6-8939-770401241bf0" (UID: "adc797dd-c98b-4af6-8939-770401241bf0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:03:26.946494 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.946468 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-workload-certs" pod "adc797dd-c98b-4af6-8939-770401241bf0" (UID: "adc797dd-c98b-4af6-8939-770401241bf0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:03:26.946838 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.946816 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-workload-socket" pod "adc797dd-c98b-4af6-8939-770401241bf0" (UID: "adc797dd-c98b-4af6-8939-770401241bf0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:03:26.947151 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.947130 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:03:26.947306 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.947291 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adc797dd-c98b-4af6-8939-770401241bf0-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:03:26.947411 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.947399 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:03:26.947494 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.947484 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:03:26.947587 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.947576 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:03:26.948946 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.948911 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adc797dd-c98b-4af6-8939-770401241bf0-istio-podinfo" pod "adc797dd-c98b-4af6-8939-770401241bf0" (UID: "adc797dd-c98b-4af6-8939-770401241bf0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:03:26.949285 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.949255 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc797dd-c98b-4af6-8939-770401241bf0-kube-api-access-kc5gv" pod "adc797dd-c98b-4af6-8939-770401241bf0" (UID: "adc797dd-c98b-4af6-8939-770401241bf0"). InnerVolumeSpecName "kube-api-access-kc5gv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:03:26.949418 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.949394 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-istio-envoy" pod "adc797dd-c98b-4af6-8939-770401241bf0" (UID: "adc797dd-c98b-4af6-8939-770401241bf0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:03:26.949484 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:26.949462 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc797dd-c98b-4af6-8939-770401241bf0-istio-token" pod "adc797dd-c98b-4af6-8939-770401241bf0" (UID: "adc797dd-c98b-4af6-8939-770401241bf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:03:27.079610 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:27.079062 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adc797dd-c98b-4af6-8939-770401241bf0-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:03:27.079610 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:27.079110 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc5gv\" (UniqueName: \"kubernetes.io/projected/adc797dd-c98b-4af6-8939-770401241bf0-kube-api-access-kc5gv\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:03:27.079610 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:27.079125 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adc797dd-c98b-4af6-8939-770401241bf0-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:03:27.079610 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:27.079139 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adc797dd-c98b-4af6-8939-770401241bf0-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:03:27.782245 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:27.782213 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" Nov 03 17:03:27.782774 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:27.782275 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p" event={"ID":"adc797dd-c98b-4af6-8939-770401241bf0","Type":"ContainerDied","Data":"046c349016d0a4e10a2b19ecfb34838ba895af90b392fba6644c63841855147e"} Nov 03 17:03:27.782774 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:27.782331 4947 scope.go:122] "RemoveContainer" containerID="c9bb15903a12cd75d6e3d2abd0c2ddd427a4bf156735c531f92a2dfa3335bc54" Nov 03 17:03:27.854061 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:27.854026 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p"] Nov 03 17:03:27.867256 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:27.867221 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-rrl8p"] Nov 03 17:03:28.567249 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:28.567212 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adc797dd-c98b-4af6-8939-770401241bf0" path="/var/lib/kubelet/pods/adc797dd-c98b-4af6-8939-770401241bf0/volumes" Nov 03 17:03:30.133734 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:03:30.133668 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:03:48.734044 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:48.733096 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": read tcp 10.132.0.2:56766->10.132.0.186:15021: read: connection reset by peer" start-of-body= Nov 03 17:03:48.734044 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:48.733188 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" podUID="5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": read tcp 10.132.0.2:56766->10.132.0.186:15021: read: connection reset by peer" Nov 03 17:03:49.111210 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:49.111177 4947 generic.go:356] "Generic (PLEG): container finished" podID="5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" containerID="ddc9ab561808b41b10a6fe5eff8401fde77078b66c59bb58fd9579b47172a5f7" exitCode=0 Nov 03 17:03:49.111390 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:49.111251 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" event={"ID":"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e","Type":"ContainerDied","Data":"ddc9ab561808b41b10a6fe5eff8401fde77078b66c59bb58fd9579b47172a5f7"} Nov 03 17:03:49.111831 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:49.111815 4947 scope.go:122] "RemoveContainer" containerID="ddc9ab561808b41b10a6fe5eff8401fde77078b66c59bb58fd9579b47172a5f7" Nov 03 17:03:49.113817 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:49.113787 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:03:49.113892 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:49.113860 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:03:49.113933 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:49.113893 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:03:50.120853 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:50.120806 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" event={"ID":"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e","Type":"ContainerStarted","Data":"d864ad35936a5425a0ad6caff3dd7b24ba87e7fc952a1787c0d75363f873e1ae"} Nov 03 17:03:50.927602 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:50.927550 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:03:50.928069 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:50.928038 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 03 17:03:50.928188 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:50.928102 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" podUID="5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 03 17:03:51.929110 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:51.929076 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 03 17:03:51.929579 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:51.929133 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" podUID="5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 03 17:03:52.928213 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:52.928176 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 03 17:03:52.928446 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:52.928240 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" podUID="5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 03 17:03:53.928014 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:53.927977 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 03 17:03:53.928802 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:53.928768 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" podUID="5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 03 17:03:54.928404 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:54.928370 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 03 17:03:54.928910 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:54.928882 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" podUID="5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 03 17:03:55.251774 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.251734 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:03:55.691117 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.691033 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z"] Nov 03 17:03:55.691833 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.691818 4947 cpu_manager.go:414] "containerMap: removing container" podUID="adc797dd-c98b-4af6-8939-770401241bf0" containerName="istio-proxy" Nov 03 17:03:55.691884 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.691835 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc797dd-c98b-4af6-8939-770401241bf0" containerName="istio-proxy" Nov 03 17:03:55.691884 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.691854 4947 cpu_manager.go:414] "containerMap: removing container" podUID="adc797dd-c98b-4af6-8939-770401241bf0" containerName="istio-proxy" Nov 03 17:03:55.691884 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.691860 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adc797dd-c98b-4af6-8939-770401241bf0" containerName="istio-proxy" Nov 03 17:03:55.692011 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.692002 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc797dd-c98b-4af6-8939-770401241bf0" containerName="istio-proxy" Nov 03 17:03:55.697129 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.697104 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" Nov 03 17:03:55.730032 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.729251 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"payload-processing-external-model-e2e--f8f5d575-dockercfg-5tk6w\"" Nov 03 17:03:55.730032 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.729352 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"openshift-service-ca.crt\"" Nov 03 17:03:55.730032 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.729392 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"agc-cred-8bf992ef4bf9\"" Nov 03 17:03:55.730379 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.730352 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"routing-overlay\"" Nov 03 17:03:55.730662 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.730642 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fdbf0c\"" Nov 03 17:03:55.730860 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.730844 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"kube-root-ca.crt\"" Nov 03 17:03:55.732639 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.732621 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z"] Nov 03 17:03:55.784441 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.784399 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjwkh\" (UniqueName: \"kubernetes.io/projected/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-kube-api-access-mjwkh\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z\" (UID: \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" Nov 03 17:03:55.784840 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.784819 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z\" (UID: \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" Nov 03 17:03:55.784998 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.784987 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z\" (UID: \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" Nov 03 17:03:55.785144 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.785133 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z\" (UID: \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" Nov 03 17:03:55.903729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.887215 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjwkh\" (UniqueName: \"kubernetes.io/projected/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-kube-api-access-mjwkh\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z\" (UID: \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" Nov 03 17:03:55.903729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.887373 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z\" (UID: \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" Nov 03 17:03:55.903729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.887440 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z\" (UID: \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" Nov 03 17:03:55.903729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.887617 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z\" (UID: \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" Nov 03 17:03:55.903729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.888577 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z\" (UID: \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" Nov 03 17:03:55.926742 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.914340 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z\" (UID: \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" Nov 03 17:03:55.946660 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.945466 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Nov 03 17:03:55.946660 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.945527 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" podUID="5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Nov 03 17:03:55.946660 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.946550 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjwkh\" (UniqueName: \"kubernetes.io/projected/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-kube-api-access-mjwkh\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z\" (UID: \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" Nov 03 17:03:55.960735 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:55.947642 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z\" (UID: \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" Nov 03 17:03:56.052100 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:56.052043 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" Nov 03 17:03:56.764113 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:03:56.764079 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ae9bb6_1d56_4007_afc6_9a4b9c4bcb18.slice/crio-a9c54e1f3d7a375d05dce68bcb082d121ee08c746feaa5d532801ab83c993e96 WatchSource:0}: Error finding container a9c54e1f3d7a375d05dce68bcb082d121ee08c746feaa5d532801ab83c993e96: Status 404 returned error can't find the container with id a9c54e1f3d7a375d05dce68bcb082d121ee08c746feaa5d532801ab83c993e96 Nov 03 17:03:56.766723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:56.766673 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z"] Nov 03 17:03:57.035530 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:57.035440 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:03:57.036104 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:57.035834 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:03:57.036603 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:57.036582 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:03:57.217854 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:57.217816 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" event={"ID":"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18","Type":"ContainerStarted","Data":"a7e01b7c0f57d7f5c04ed188f9bcd5254fd675384db74d7e257c0c9cb9b71372"} Nov 03 17:03:57.218032 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:57.217877 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" event={"ID":"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18","Type":"ContainerStarted","Data":"a9c54e1f3d7a375d05dce68bcb082d121ee08c746feaa5d532801ab83c993e96"} Nov 03 17:03:57.273851 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:57.273798 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" podStartSLOduration=2.273782653 podStartE2EDuration="2.273782653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:03:57.27195209 +0000 UTC m=+2980.860952062" watchObservedRunningTime="2026-11-03 17:03:57.273782653 +0000 UTC m=+2980.862782622" Nov 03 17:03:58.225922 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:03:58.225890 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" Nov 03 17:04:00.132136 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:04:00.132101 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:04:09.237005 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:09.236974 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" Nov 03 17:04:16.998735 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:16.998670 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:04:17.002715 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:17.002671 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:04:21.924548 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:21.924508 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c"] Nov 03 17:04:21.925619 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:21.925596 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="adc797dd-c98b-4af6-8939-770401241bf0" containerName="istio-proxy" Nov 03 17:04:21.931031 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:21.931008 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" Nov 03 17:04:21.960736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:21.960668 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"agc-cred-f09ac2317b26\"" Nov 03 17:04:21.967800 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:21.967756 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c"] Nov 03 17:04:21.978009 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:21.977967 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2733a9d-062f-4ea5-927b-97d974df3f92-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c\" (UID: \"e2733a9d-062f-4ea5-927b-97d974df3f92\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" Nov 03 17:04:21.978221 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:21.978093 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rc78\" (UniqueName: \"kubernetes.io/projected/e2733a9d-062f-4ea5-927b-97d974df3f92-kube-api-access-7rc78\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c\" (UID: \"e2733a9d-062f-4ea5-927b-97d974df3f92\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" Nov 03 17:04:21.978387 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:21.978365 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2733a9d-062f-4ea5-927b-97d974df3f92-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c\" (UID: \"e2733a9d-062f-4ea5-927b-97d974df3f92\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" Nov 03 17:04:21.978564 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:21.978547 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2733a9d-062f-4ea5-927b-97d974df3f92-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c\" (UID: \"e2733a9d-062f-4ea5-927b-97d974df3f92\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" Nov 03 17:04:22.079798 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:22.079762 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2733a9d-062f-4ea5-927b-97d974df3f92-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c\" (UID: \"e2733a9d-062f-4ea5-927b-97d974df3f92\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" Nov 03 17:04:22.080036 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:22.079912 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2733a9d-062f-4ea5-927b-97d974df3f92-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c\" (UID: \"e2733a9d-062f-4ea5-927b-97d974df3f92\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" Nov 03 17:04:22.080036 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:22.079965 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2733a9d-062f-4ea5-927b-97d974df3f92-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c\" (UID: \"e2733a9d-062f-4ea5-927b-97d974df3f92\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" Nov 03 17:04:22.080036 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:22.080010 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rc78\" (UniqueName: \"kubernetes.io/projected/e2733a9d-062f-4ea5-927b-97d974df3f92-kube-api-access-7rc78\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c\" (UID: \"e2733a9d-062f-4ea5-927b-97d974df3f92\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" Nov 03 17:04:22.080665 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:22.080636 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2733a9d-062f-4ea5-927b-97d974df3f92-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c\" (UID: \"e2733a9d-062f-4ea5-927b-97d974df3f92\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" Nov 03 17:04:22.080878 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:22.080855 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2733a9d-062f-4ea5-927b-97d974df3f92-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c\" (UID: \"e2733a9d-062f-4ea5-927b-97d974df3f92\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" Nov 03 17:04:22.082711 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:22.082678 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2733a9d-062f-4ea5-927b-97d974df3f92-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c\" (UID: \"e2733a9d-062f-4ea5-927b-97d974df3f92\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" Nov 03 17:04:22.122429 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:22.122394 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rc78\" (UniqueName: \"kubernetes.io/projected/e2733a9d-062f-4ea5-927b-97d974df3f92-kube-api-access-7rc78\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c\" (UID: \"e2733a9d-062f-4ea5-927b-97d974df3f92\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" Nov 03 17:04:22.248442 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:22.247048 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" Nov 03 17:04:22.573121 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:04:22.573089 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2733a9d_062f_4ea5_927b_97d974df3f92.slice/crio-3af94830b54a4fec6297522993fb8df36f6285d2783e9938afc7a43f3356d785 WatchSource:0}: Error finding container 3af94830b54a4fec6297522993fb8df36f6285d2783e9938afc7a43f3356d785: Status 404 returned error can't find the container with id 3af94830b54a4fec6297522993fb8df36f6285d2783e9938afc7a43f3356d785 Nov 03 17:04:22.573605 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:22.573580 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c"] Nov 03 17:04:22.591911 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:22.591876 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" event={"ID":"e2733a9d-062f-4ea5-927b-97d974df3f92","Type":"ContainerStarted","Data":"3af94830b54a4fec6297522993fb8df36f6285d2783e9938afc7a43f3356d785"} Nov 03 17:04:23.602135 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:23.602097 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" event={"ID":"e2733a9d-062f-4ea5-927b-97d974df3f92","Type":"ContainerStarted","Data":"312a2201bb3a642cc9792f967243e2885671c3b0479aa37e31257ff55e79ea42"} Nov 03 17:04:23.602603 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:23.602285 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" Nov 03 17:04:23.647810 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:23.647750 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" podStartSLOduration=2.647734766 podStartE2EDuration="2.647734766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:04:23.646922255 +0000 UTC m=+3007.235922224" watchObservedRunningTime="2026-11-03 17:04:23.647734766 +0000 UTC m=+3007.236734731" Nov 03 17:04:30.136952 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:04:30.136853 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:04:34.614190 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:34.614161 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" Nov 03 17:04:34.694519 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:34.694468 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw"] Nov 03 17:04:35.360183 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:35.360094 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:04:39.705737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:39.698098 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" podUID="61811daa-ebc2-41b4-a703-dfbf66f54ed2" containerName="payload-processing" containerID="cri-o://f677f506f04ed33b1211e8bd9972ab6a67e8f0e2e244013af592ea5d9b3ec1ef" gracePeriod=25 Nov 03 17:04:39.890288 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:04:39.887188 4947 logging.go:55] [core] [Channel #3494 SubChannel #3495]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.188:50052", ServerName: "10.132.0.188:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.188:50052: connect: connection refused" Nov 03 17:04:40.027188 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.027153 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" Nov 03 17:04:40.135906 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.135864 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w"] Nov 03 17:04:40.136803 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.136782 4947 cpu_manager.go:414] "containerMap: removing container" podUID="61811daa-ebc2-41b4-a703-dfbf66f54ed2" containerName="payload-processing" Nov 03 17:04:40.136879 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.136807 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61811daa-ebc2-41b4-a703-dfbf66f54ed2" containerName="payload-processing" Nov 03 17:04:40.137141 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.137117 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="61811daa-ebc2-41b4-a703-dfbf66f54ed2" containerName="payload-processing" Nov 03 17:04:40.141324 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.141301 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" Nov 03 17:04:40.157718 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.157660 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"agc-cred-76c8387ff3f3\"" Nov 03 17:04:40.157718 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.157688 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"payload-processing-external-model-e2e--d0a79461-dockercfg-c9d8c\"" Nov 03 17:04:40.158012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.157661 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"kube-root-ca.crt\"" Nov 03 17:04:40.158012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.157796 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"agc-cred-4ce9c9bbf166\"" Nov 03 17:04:40.158012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.157809 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ca9d7c\"" Nov 03 17:04:40.158012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.157815 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"openshift-service-ca.crt\"" Nov 03 17:04:40.158012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.157833 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"routing-overlay\"" Nov 03 17:04:40.158012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.157940 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61811daa-ebc2-41b4-a703-dfbf66f54ed2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/61811daa-ebc2-41b4-a703-dfbf66f54ed2-provider-credentials\") pod \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\" (UID: \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\") " Nov 03 17:04:40.158327 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.158050 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61811daa-ebc2-41b4-a703-dfbf66f54ed2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61811daa-ebc2-41b4-a703-dfbf66f54ed2-plugins-config-volume\") pod \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\" (UID: \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\") " Nov 03 17:04:40.158327 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.158134 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61811daa-ebc2-41b4-a703-dfbf66f54ed2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61811daa-ebc2-41b4-a703-dfbf66f54ed2-routing-overlay-volume\") pod \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\" (UID: \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\") " Nov 03 17:04:40.158442 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.158342 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61811daa-ebc2-41b4-a703-dfbf66f54ed2-kube-api-access-f9zt9\" (UniqueName: \"kubernetes.io/projected/61811daa-ebc2-41b4-a703-dfbf66f54ed2-kube-api-access-f9zt9\") pod \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\" (UID: \"61811daa-ebc2-41b4-a703-dfbf66f54ed2\") " Nov 03 17:04:40.158561 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.158508 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61811daa-ebc2-41b4-a703-dfbf66f54ed2-plugins-config-volume" pod "61811daa-ebc2-41b4-a703-dfbf66f54ed2" (UID: "61811daa-ebc2-41b4-a703-dfbf66f54ed2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:04:40.158561 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.158515 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61811daa-ebc2-41b4-a703-dfbf66f54ed2-routing-overlay-volume" pod "61811daa-ebc2-41b4-a703-dfbf66f54ed2" (UID: "61811daa-ebc2-41b4-a703-dfbf66f54ed2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:04:40.159009 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.158984 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61811daa-ebc2-41b4-a703-dfbf66f54ed2-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:04:40.159009 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.159005 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61811daa-ebc2-41b4-a703-dfbf66f54ed2-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:04:40.160621 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.160591 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61811daa-ebc2-41b4-a703-dfbf66f54ed2-provider-credentials" pod "61811daa-ebc2-41b4-a703-dfbf66f54ed2" (UID: "61811daa-ebc2-41b4-a703-dfbf66f54ed2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:04:40.161136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.161113 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61811daa-ebc2-41b4-a703-dfbf66f54ed2-kube-api-access-f9zt9" pod "61811daa-ebc2-41b4-a703-dfbf66f54ed2" (UID: "61811daa-ebc2-41b4-a703-dfbf66f54ed2"). InnerVolumeSpecName "kube-api-access-f9zt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:04:40.161632 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.161606 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w"] Nov 03 17:04:40.260833 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.260799 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx6z4\" (UniqueName: \"kubernetes.io/projected/bc89793f-a6fb-4546-a4bf-948a747e66fa-kube-api-access-lx6z4\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w\" (UID: \"bc89793f-a6fb-4546-a4bf-948a747e66fa\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" Nov 03 17:04:40.261208 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.261179 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc89793f-a6fb-4546-a4bf-948a747e66fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w\" (UID: \"bc89793f-a6fb-4546-a4bf-948a747e66fa\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" Nov 03 17:04:40.261338 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.261314 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc89793f-a6fb-4546-a4bf-948a747e66fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w\" (UID: \"bc89793f-a6fb-4546-a4bf-948a747e66fa\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" Nov 03 17:04:40.261453 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.261437 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc89793f-a6fb-4546-a4bf-948a747e66fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w\" (UID: \"bc89793f-a6fb-4546-a4bf-948a747e66fa\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" Nov 03 17:04:40.261526 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.261515 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61811daa-ebc2-41b4-a703-dfbf66f54ed2-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:04:40.261561 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.261531 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9zt9\" (UniqueName: \"kubernetes.io/projected/61811daa-ebc2-41b4-a703-dfbf66f54ed2-kube-api-access-f9zt9\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:04:40.363348 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.363310 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc89793f-a6fb-4546-a4bf-948a747e66fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w\" (UID: \"bc89793f-a6fb-4546-a4bf-948a747e66fa\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" Nov 03 17:04:40.363575 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.363384 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc89793f-a6fb-4546-a4bf-948a747e66fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w\" (UID: \"bc89793f-a6fb-4546-a4bf-948a747e66fa\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" Nov 03 17:04:40.363575 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.363450 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc89793f-a6fb-4546-a4bf-948a747e66fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w\" (UID: \"bc89793f-a6fb-4546-a4bf-948a747e66fa\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" Nov 03 17:04:40.363575 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.363490 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx6z4\" (UniqueName: \"kubernetes.io/projected/bc89793f-a6fb-4546-a4bf-948a747e66fa-kube-api-access-lx6z4\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w\" (UID: \"bc89793f-a6fb-4546-a4bf-948a747e66fa\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" Nov 03 17:04:40.364274 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.364242 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc89793f-a6fb-4546-a4bf-948a747e66fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w\" (UID: \"bc89793f-a6fb-4546-a4bf-948a747e66fa\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" Nov 03 17:04:40.364379 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.364310 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc89793f-a6fb-4546-a4bf-948a747e66fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w\" (UID: \"bc89793f-a6fb-4546-a4bf-948a747e66fa\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" Nov 03 17:04:40.366792 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.366767 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc89793f-a6fb-4546-a4bf-948a747e66fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w\" (UID: \"bc89793f-a6fb-4546-a4bf-948a747e66fa\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" Nov 03 17:04:40.388500 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.388463 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx6z4\" (UniqueName: \"kubernetes.io/projected/bc89793f-a6fb-4546-a4bf-948a747e66fa-kube-api-access-lx6z4\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w\" (UID: \"bc89793f-a6fb-4546-a4bf-948a747e66fa\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" Nov 03 17:04:40.459320 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.459278 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" Nov 03 17:04:40.767830 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.767795 4947 generic.go:356] "Generic (PLEG): container finished" podID="61811daa-ebc2-41b4-a703-dfbf66f54ed2" containerID="f677f506f04ed33b1211e8bd9972ab6a67e8f0e2e244013af592ea5d9b3ec1ef" exitCode=0 Nov 03 17:04:40.768351 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.767846 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" Nov 03 17:04:40.768351 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.767890 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" event={"ID":"61811daa-ebc2-41b4-a703-dfbf66f54ed2","Type":"ContainerDied","Data":"f677f506f04ed33b1211e8bd9972ab6a67e8f0e2e244013af592ea5d9b3ec1ef"} Nov 03 17:04:40.768351 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.767984 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" event={"ID":"61811daa-ebc2-41b4-a703-dfbf66f54ed2","Type":"ContainerDied","Data":"c9702011e02ba0bc40e842e5265caa944c654f7bd3d9106323ab07c067efc003"} Nov 03 17:04:40.768351 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.768022 4947 scope.go:122] "RemoveContainer" containerID="f677f506f04ed33b1211e8bd9972ab6a67e8f0e2e244013af592ea5d9b3ec1ef" Nov 03 17:04:40.779249 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:04:40.779181 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc89793f_a6fb_4546_a4bf_948a747e66fa.slice/crio-ae18559cc8b1a1f327da9c48b88e14c891c07fdf29803b80b19a3418612dc384 WatchSource:0}: Error finding container ae18559cc8b1a1f327da9c48b88e14c891c07fdf29803b80b19a3418612dc384: Status 404 returned error can't find the container with id ae18559cc8b1a1f327da9c48b88e14c891c07fdf29803b80b19a3418612dc384 Nov 03 17:04:40.779381 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.779307 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w"] Nov 03 17:04:40.783114 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.783090 4947 scope.go:122] "RemoveContainer" containerID="f677f506f04ed33b1211e8bd9972ab6a67e8f0e2e244013af592ea5d9b3ec1ef" Nov 03 17:04:40.783466 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:04:40.783431 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f677f506f04ed33b1211e8bd9972ab6a67e8f0e2e244013af592ea5d9b3ec1ef\": container with ID starting with f677f506f04ed33b1211e8bd9972ab6a67e8f0e2e244013af592ea5d9b3ec1ef not found: ID does not exist" containerID="f677f506f04ed33b1211e8bd9972ab6a67e8f0e2e244013af592ea5d9b3ec1ef" Nov 03 17:04:40.783587 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.783472 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f677f506f04ed33b1211e8bd9972ab6a67e8f0e2e244013af592ea5d9b3ec1ef"} err="failed to get container status \"f677f506f04ed33b1211e8bd9972ab6a67e8f0e2e244013af592ea5d9b3ec1ef\": rpc error: code = NotFound desc = could not find container \"f677f506f04ed33b1211e8bd9972ab6a67e8f0e2e244013af592ea5d9b3ec1ef\": container with ID starting with f677f506f04ed33b1211e8bd9972ab6a67e8f0e2e244013af592ea5d9b3ec1ef not found: ID does not exist" Nov 03 17:04:40.827828 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.827792 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw"] Nov 03 17:04:40.832527 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.832498 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw"] Nov 03 17:04:40.887650 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:40.887606 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b678gknw" podUID="61811daa-ebc2-41b4-a703-dfbf66f54ed2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.188:50052\" within 1s: context deadline exceeded" Nov 03 17:04:40.887880 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:04:40.887741 4947 logging.go:55] [core] [Channel #3494 SubChannel #3495]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.188:50052", ServerName: "10.132.0.188:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.188:50052: operation was canceled" Nov 03 17:04:41.816800 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:41.809899 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" event={"ID":"bc89793f-a6fb-4546-a4bf-948a747e66fa","Type":"ContainerStarted","Data":"40d2ef5fecaf82e8691c1fa5530d30d61704b6c5f4bfe8434797bb26f62286d8"} Nov 03 17:04:41.816800 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:41.810015 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" event={"ID":"bc89793f-a6fb-4546-a4bf-948a747e66fa","Type":"ContainerStarted","Data":"ae18559cc8b1a1f327da9c48b88e14c891c07fdf29803b80b19a3418612dc384"} Nov 03 17:04:41.816800 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:41.813822 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" Nov 03 17:04:41.866692 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:41.866636 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" podStartSLOduration=1.866622405 podStartE2EDuration="1.866622405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:04:41.865408229 +0000 UTC m=+3025.454408198" watchObservedRunningTime="2026-11-03 17:04:41.866622405 +0000 UTC m=+3025.455622374" Nov 03 17:04:42.569543 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:42.569509 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61811daa-ebc2-41b4-a703-dfbf66f54ed2" path="/var/lib/kubelet/pods/61811daa-ebc2-41b4-a703-dfbf66f54ed2/volumes" Nov 03 17:04:53.829458 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:04:53.829428 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" Nov 03 17:05:00.187028 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:05:00.186986 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:05:09.733762 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:09.729797 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:05:09.966737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:09.964519 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:05:18.171489 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:18.171448 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv"] Nov 03 17:05:18.177359 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:18.177334 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" Nov 03 17:05:18.198878 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:18.198847 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv"] Nov 03 17:05:18.206379 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:18.206349 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"agc-cred-b833a8f9cdcf\"" Nov 03 17:05:18.262007 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:18.261950 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49219a67-6c62-444a-aaff-4501ec769c84-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv\" (UID: \"49219a67-6c62-444a-aaff-4501ec769c84\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" Nov 03 17:05:18.262227 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:18.262152 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv\" (UID: \"49219a67-6c62-444a-aaff-4501ec769c84\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" Nov 03 17:05:18.262282 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:18.262258 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7km2h\" (UniqueName: \"kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-kube-api-access-7km2h\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv\" (UID: \"49219a67-6c62-444a-aaff-4501ec769c84\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" Nov 03 17:05:18.262353 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:18.262341 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49219a67-6c62-444a-aaff-4501ec769c84-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv\" (UID: \"49219a67-6c62-444a-aaff-4501ec769c84\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" Nov 03 17:05:18.363872 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:18.363837 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49219a67-6c62-444a-aaff-4501ec769c84-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv\" (UID: \"49219a67-6c62-444a-aaff-4501ec769c84\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" Nov 03 17:05:18.364154 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:18.363976 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv\" (UID: \"49219a67-6c62-444a-aaff-4501ec769c84\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" Nov 03 17:05:18.364154 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:18.364082 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7km2h\" (UniqueName: \"kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-kube-api-access-7km2h\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv\" (UID: \"49219a67-6c62-444a-aaff-4501ec769c84\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" Nov 03 17:05:18.364299 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:18.364184 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49219a67-6c62-444a-aaff-4501ec769c84-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv\" (UID: \"49219a67-6c62-444a-aaff-4501ec769c84\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" Nov 03 17:05:18.364677 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:18.364648 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49219a67-6c62-444a-aaff-4501ec769c84-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv\" (UID: \"49219a67-6c62-444a-aaff-4501ec769c84\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" Nov 03 17:05:18.365022 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:18.364994 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49219a67-6c62-444a-aaff-4501ec769c84-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv\" (UID: \"49219a67-6c62-444a-aaff-4501ec769c84\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" Nov 03 17:05:18.366927 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:18.366902 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv\" (UID: \"49219a67-6c62-444a-aaff-4501ec769c84\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" Nov 03 17:05:18.388591 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:18.388559 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7km2h\" (UniqueName: \"kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-kube-api-access-7km2h\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv\" (UID: \"49219a67-6c62-444a-aaff-4501ec769c84\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" Nov 03 17:05:18.492111 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:18.492073 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" Nov 03 17:05:18.808053 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:05:18.808014 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49219a67_6c62_444a_aaff_4501ec769c84.slice/crio-6b617c23f08a67c3496ebe8a931b495a54108d64ccd7cf5de1343771116b9de9 WatchSource:0}: Error finding container 6b617c23f08a67c3496ebe8a931b495a54108d64ccd7cf5de1343771116b9de9: Status 404 returned error can't find the container with id 6b617c23f08a67c3496ebe8a931b495a54108d64ccd7cf5de1343771116b9de9 Nov 03 17:05:18.809255 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:18.809227 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv"] Nov 03 17:05:19.450858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:19.450812 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" event={"ID":"49219a67-6c62-444a-aaff-4501ec769c84","Type":"ContainerStarted","Data":"c6b5a9fc009a76cd550353cb207c95695829f4cf1d0f68c9f473037a52990e98"} Nov 03 17:05:19.451270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:19.450906 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" event={"ID":"49219a67-6c62-444a-aaff-4501ec769c84","Type":"ContainerStarted","Data":"6b617c23f08a67c3496ebe8a931b495a54108d64ccd7cf5de1343771116b9de9"} Nov 03 17:05:19.451474 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:19.451435 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" Nov 03 17:05:19.497662 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:19.497608 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" podStartSLOduration=1.497593484 podStartE2EDuration="1.497593484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:05:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:05:19.495817422 +0000 UTC m=+3063.084817391" watchObservedRunningTime="2026-11-03 17:05:19.497593484 +0000 UTC m=+3063.086593453" Nov 03 17:05:21.543933 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.543900 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g"] Nov 03 17:05:21.548886 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.548864 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" Nov 03 17:05:21.569492 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.569450 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g\" (UID: \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" Nov 03 17:05:21.569797 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.569775 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g\" (UID: \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" Nov 03 17:05:21.569944 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.569923 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g\" (UID: \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" Nov 03 17:05:21.570121 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.570098 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk4w4\" (UniqueName: \"kubernetes.io/projected/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-kube-api-access-vk4w4\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g\" (UID: \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" Nov 03 17:05:21.572744 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.572715 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g"] Nov 03 17:05:21.578912 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.578869 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a0e27\"/\"kube-root-ca.crt\"" Nov 03 17:05:21.578912 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.578880 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a0e27\"/\"payload-processing-external-model-plugins-e2e-extmod-a-3a0e27\"" Nov 03 17:05:21.579133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.579038 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a0e27\"/\"payload-processing-external-model-e2e--8dd308cd-dockercfg-dmdmk\"" Nov 03 17:05:21.579133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.579040 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a0e27\"/\"routing-overlay\"" Nov 03 17:05:21.582316 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.582281 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a0e27\"/\"e2e-cmp-legacy-883f56-provider-cred\"" Nov 03 17:05:21.582459 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.582347 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a0e27\"/\"openshift-service-ca.crt\"" Nov 03 17:05:21.671478 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.671440 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk4w4\" (UniqueName: \"kubernetes.io/projected/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-kube-api-access-vk4w4\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g\" (UID: \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" Nov 03 17:05:21.671717 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.671510 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g\" (UID: \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" Nov 03 17:05:21.671717 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.671643 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g\" (UID: \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" Nov 03 17:05:21.671901 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.671718 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g\" (UID: \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" Nov 03 17:05:21.672344 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.672321 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g\" (UID: \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" Nov 03 17:05:21.672451 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.672396 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g\" (UID: \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" Nov 03 17:05:21.674338 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.674316 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g\" (UID: \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" Nov 03 17:05:21.698607 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.698575 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk4w4\" (UniqueName: \"kubernetes.io/projected/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-kube-api-access-vk4w4\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g\" (UID: \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" Nov 03 17:05:21.868320 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:21.868237 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" Nov 03 17:05:22.428925 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:05:22.428891 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42eb2139_5c0f_47f3_8c63_dc38a2ff85de.slice/crio-a630c4dfc22ecdc51a06cc48ddec05e7127bd2a9cbc3ef1672cf413d1186f888 WatchSource:0}: Error finding container a630c4dfc22ecdc51a06cc48ddec05e7127bd2a9cbc3ef1672cf413d1186f888: Status 404 returned error can't find the container with id a630c4dfc22ecdc51a06cc48ddec05e7127bd2a9cbc3ef1672cf413d1186f888 Nov 03 17:05:22.429768 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:22.429747 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g"] Nov 03 17:05:22.478060 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:22.478021 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" event={"ID":"42eb2139-5c0f-47f3-8c63-dc38a2ff85de","Type":"ContainerStarted","Data":"a630c4dfc22ecdc51a06cc48ddec05e7127bd2a9cbc3ef1672cf413d1186f888"} Nov 03 17:05:23.498870 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:23.498838 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" event={"ID":"42eb2139-5c0f-47f3-8c63-dc38a2ff85de","Type":"ContainerStarted","Data":"4f8cf3a1f86c3b1b8c733d6c5ef470cda822437631c7a93c853cab1076b43189"} Nov 03 17:05:23.499282 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:23.498914 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" Nov 03 17:05:23.550414 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:23.550356 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" podStartSLOduration=2.550338644 podStartE2EDuration="2.550338644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:05:23.547219412 +0000 UTC m=+3067.136219380" watchObservedRunningTime="2026-11-03 17:05:23.550338644 +0000 UTC m=+3067.139338615" Nov 03 17:05:30.132948 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:05:30.132906 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:05:30.462917 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:30.462829 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" Nov 03 17:05:30.544146 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:30.544103 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c"] Nov 03 17:05:32.220111 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:32.220076 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z"] Nov 03 17:05:33.355577 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:33.349525 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:05:34.510563 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:34.510528 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" Nov 03 17:05:34.690752 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:34.690713 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n"] Nov 03 17:05:34.691046 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:34.690995 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" podUID="c7ef9905-f189-4f71-a6b9-10add14d0088" containerName="istio-proxy" containerID="cri-o://78c1d6a34d89dc9a909741647427717b9959c6cccfde0e8d7e16c1e9e9b3406f" gracePeriod=30 Nov 03 17:05:35.545774 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:35.545163 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" podUID="e2733a9d-062f-4ea5-927b-97d974df3f92" containerName="payload-processing" containerID="cri-o://312a2201bb3a642cc9792f967243e2885671c3b0479aa37e31257ff55e79ea42" gracePeriod=25 Nov 03 17:05:35.673339 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:35.673304 4947 generic.go:356] "Generic (PLEG): container finished" podID="e2733a9d-062f-4ea5-927b-97d974df3f92" containerID="312a2201bb3a642cc9792f967243e2885671c3b0479aa37e31257ff55e79ea42" exitCode=0 Nov 03 17:05:35.673520 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:35.673378 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" event={"ID":"e2733a9d-062f-4ea5-927b-97d974df3f92","Type":"ContainerDied","Data":"312a2201bb3a642cc9792f967243e2885671c3b0479aa37e31257ff55e79ea42"} Nov 03 17:05:35.870211 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:35.870182 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" Nov 03 17:05:35.955898 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:35.955860 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2733a9d-062f-4ea5-927b-97d974df3f92-kube-api-access-7rc78\" (UniqueName: \"kubernetes.io/projected/e2733a9d-062f-4ea5-927b-97d974df3f92-kube-api-access-7rc78\") pod \"e2733a9d-062f-4ea5-927b-97d974df3f92\" (UID: \"e2733a9d-062f-4ea5-927b-97d974df3f92\") " Nov 03 17:05:35.956094 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:35.955993 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2733a9d-062f-4ea5-927b-97d974df3f92-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2733a9d-062f-4ea5-927b-97d974df3f92-provider-credentials\") pod \"e2733a9d-062f-4ea5-927b-97d974df3f92\" (UID: \"e2733a9d-062f-4ea5-927b-97d974df3f92\") " Nov 03 17:05:35.956145 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:35.956102 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2733a9d-062f-4ea5-927b-97d974df3f92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2733a9d-062f-4ea5-927b-97d974df3f92-plugins-config-volume\") pod \"e2733a9d-062f-4ea5-927b-97d974df3f92\" (UID: \"e2733a9d-062f-4ea5-927b-97d974df3f92\") " Nov 03 17:05:35.956310 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:35.956283 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2733a9d-062f-4ea5-927b-97d974df3f92-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2733a9d-062f-4ea5-927b-97d974df3f92-routing-overlay-volume\") pod \"e2733a9d-062f-4ea5-927b-97d974df3f92\" (UID: \"e2733a9d-062f-4ea5-927b-97d974df3f92\") " Nov 03 17:05:35.956643 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:35.956621 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2733a9d-062f-4ea5-927b-97d974df3f92-plugins-config-volume" pod "e2733a9d-062f-4ea5-927b-97d974df3f92" (UID: "e2733a9d-062f-4ea5-927b-97d974df3f92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:05:35.956765 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:35.956669 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2733a9d-062f-4ea5-927b-97d974df3f92-routing-overlay-volume" pod "e2733a9d-062f-4ea5-927b-97d974df3f92" (UID: "e2733a9d-062f-4ea5-927b-97d974df3f92"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:05:35.957412 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:35.957391 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2733a9d-062f-4ea5-927b-97d974df3f92-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:35.957472 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:35.957421 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2733a9d-062f-4ea5-927b-97d974df3f92-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:35.958262 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:35.958230 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2733a9d-062f-4ea5-927b-97d974df3f92-kube-api-access-7rc78" pod "e2733a9d-062f-4ea5-927b-97d974df3f92" (UID: "e2733a9d-062f-4ea5-927b-97d974df3f92"). InnerVolumeSpecName "kube-api-access-7rc78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:35.958530 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:35.958505 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2733a9d-062f-4ea5-927b-97d974df3f92-provider-credentials" pod "e2733a9d-062f-4ea5-927b-97d974df3f92" (UID: "e2733a9d-062f-4ea5-927b-97d974df3f92"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:36.058607 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:36.058560 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2733a9d-062f-4ea5-927b-97d974df3f92-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:36.058607 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:36.058594 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rc78\" (UniqueName: \"kubernetes.io/projected/e2733a9d-062f-4ea5-927b-97d974df3f92-kube-api-access-7rc78\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:36.226069 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:36.225983 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf"] Nov 03 17:05:36.226241 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:36.226208 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" podUID="d281375d-bb6c-4ac6-b2fb-8bd4cb476178" containerName="istio-proxy" containerID="cri-o://58d3051d9350dc8cc3494e1eeb3fc792c8029a29bcf41f3d3614d697e83ffd7e" gracePeriod=30 Nov 03 17:05:36.685270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:36.685228 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" event={"ID":"e2733a9d-062f-4ea5-927b-97d974df3f92","Type":"ContainerDied","Data":"3af94830b54a4fec6297522993fb8df36f6285d2783e9938afc7a43f3356d785"} Nov 03 17:05:36.685835 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:36.685276 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c" Nov 03 17:05:36.685835 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:36.685334 4947 scope.go:122] "RemoveContainer" containerID="312a2201bb3a642cc9792f967243e2885671c3b0479aa37e31257ff55e79ea42" Nov 03 17:05:36.745235 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:36.745202 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c"] Nov 03 17:05:36.751946 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:36.751917 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7ffmr6c"] Nov 03 17:05:37.220616 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.220527 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" podUID="e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18" containerName="payload-processing" containerID="cri-o://a7e01b7c0f57d7f5c04ed188f9bcd5254fd675384db74d7e257c0c9cb9b71372" gracePeriod=25 Nov 03 17:05:37.516861 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.516828 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" Nov 03 17:05:37.594719 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.594669 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-provider-credentials\") pod \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\" (UID: \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\") " Nov 03 17:05:37.594993 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.594913 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-plugins-config-volume\") pod \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\" (UID: \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\") " Nov 03 17:05:37.595151 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.595125 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-routing-overlay-volume\") pod \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\" (UID: \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\") " Nov 03 17:05:37.595647 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.595172 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-kube-api-access-mjwkh\" (UniqueName: \"kubernetes.io/projected/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-kube-api-access-mjwkh\") pod \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\" (UID: \"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18\") " Nov 03 17:05:37.595647 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.595423 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-plugins-config-volume" pod "e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18" (UID: "e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:05:37.596061 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.596034 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-routing-overlay-volume" pod "e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18" (UID: "e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:05:37.596398 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.596352 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:37.596398 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.596380 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:37.598133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.598108 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-provider-credentials" pod "e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18" (UID: "e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:37.598238 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.598151 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-kube-api-access-mjwkh" pod "e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18" (UID: "e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18"). InnerVolumeSpecName "kube-api-access-mjwkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:37.696414 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.696377 4947 generic.go:356] "Generic (PLEG): container finished" podID="e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18" containerID="a7e01b7c0f57d7f5c04ed188f9bcd5254fd675384db74d7e257c0c9cb9b71372" exitCode=0 Nov 03 17:05:37.696979 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.696431 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" Nov 03 17:05:37.696979 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.696458 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" event={"ID":"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18","Type":"ContainerDied","Data":"a7e01b7c0f57d7f5c04ed188f9bcd5254fd675384db74d7e257c0c9cb9b71372"} Nov 03 17:05:37.696979 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.696515 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z" event={"ID":"e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18","Type":"ContainerDied","Data":"a9c54e1f3d7a375d05dce68bcb082d121ee08c746feaa5d532801ab83c993e96"} Nov 03 17:05:37.696979 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.696539 4947 scope.go:122] "RemoveContainer" containerID="a7e01b7c0f57d7f5c04ed188f9bcd5254fd675384db74d7e257c0c9cb9b71372" Nov 03 17:05:37.697857 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.697839 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjwkh\" (UniqueName: \"kubernetes.io/projected/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-kube-api-access-mjwkh\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:37.697951 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.697860 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:37.708163 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.708145 4947 scope.go:122] "RemoveContainer" containerID="a7e01b7c0f57d7f5c04ed188f9bcd5254fd675384db74d7e257c0c9cb9b71372" Nov 03 17:05:37.708458 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:05:37.708439 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7e01b7c0f57d7f5c04ed188f9bcd5254fd675384db74d7e257c0c9cb9b71372\": container with ID starting with a7e01b7c0f57d7f5c04ed188f9bcd5254fd675384db74d7e257c0c9cb9b71372 not found: ID does not exist" containerID="a7e01b7c0f57d7f5c04ed188f9bcd5254fd675384db74d7e257c0c9cb9b71372" Nov 03 17:05:37.708513 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.708469 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e01b7c0f57d7f5c04ed188f9bcd5254fd675384db74d7e257c0c9cb9b71372"} err="failed to get container status \"a7e01b7c0f57d7f5c04ed188f9bcd5254fd675384db74d7e257c0c9cb9b71372\": rpc error: code = NotFound desc = could not find container \"a7e01b7c0f57d7f5c04ed188f9bcd5254fd675384db74d7e257c0c9cb9b71372\": container with ID starting with a7e01b7c0f57d7f5c04ed188f9bcd5254fd675384db74d7e257c0c9cb9b71372 not found: ID does not exist" Nov 03 17:05:37.755239 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.755203 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z"] Nov 03 17:05:37.759668 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:37.759637 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-58dbrpp7z"] Nov 03 17:05:38.569004 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:38.568970 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2733a9d-062f-4ea5-927b-97d974df3f92" path="/var/lib/kubelet/pods/e2733a9d-062f-4ea5-927b-97d974df3f92/volumes" Nov 03 17:05:38.569502 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:38.569488 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18" path="/var/lib/kubelet/pods/e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18/volumes" Nov 03 17:05:38.819325 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:38.819226 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:05:38.819325 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:38.819309 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" podUID="c7ef9905-f189-4f71-a6b9-10add14d0088" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:05:39.991479 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:39.991453 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 17:05:40.142865 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.142823 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-data\") pod \"c7ef9905-f189-4f71-a6b9-10add14d0088\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " Nov 03 17:05:40.142865 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.142869 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7ef9905-f189-4f71-a6b9-10add14d0088-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ef9905-f189-4f71-a6b9-10add14d0088-istiod-ca-cert\") pod \"c7ef9905-f189-4f71-a6b9-10add14d0088\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " Nov 03 17:05:40.143125 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.142922 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-workload-socket\") pod \"c7ef9905-f189-4f71-a6b9-10add14d0088\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " Nov 03 17:05:40.143125 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.143036 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-envoy\") pod \"c7ef9905-f189-4f71-a6b9-10add14d0088\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " Nov 03 17:05:40.143235 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.143194 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ef9905-f189-4f71-a6b9-10add14d0088-kube-api-access-4dxtw\" (UniqueName: \"kubernetes.io/projected/c7ef9905-f189-4f71-a6b9-10add14d0088-kube-api-access-4dxtw\") pod \"c7ef9905-f189-4f71-a6b9-10add14d0088\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " Nov 03 17:05:40.143235 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.143165 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-data" pod "c7ef9905-f189-4f71-a6b9-10add14d0088" (UID: "c7ef9905-f189-4f71-a6b9-10add14d0088"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:05:40.143346 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.143255 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-podinfo\") pod \"c7ef9905-f189-4f71-a6b9-10add14d0088\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " Nov 03 17:05:40.143346 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.143309 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-workload-socket" pod "c7ef9905-f189-4f71-a6b9-10add14d0088" (UID: "c7ef9905-f189-4f71-a6b9-10add14d0088"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:05:40.143346 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.143330 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ef9905-f189-4f71-a6b9-10add14d0088-istiod-ca-cert" pod "c7ef9905-f189-4f71-a6b9-10add14d0088" (UID: "c7ef9905-f189-4f71-a6b9-10add14d0088"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:05:40.143506 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.143381 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-token\" (UniqueName: \"kubernetes.io/projected/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-token\") pod \"c7ef9905-f189-4f71-a6b9-10add14d0088\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " Nov 03 17:05:40.143506 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.143451 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-credential-socket\") pod \"c7ef9905-f189-4f71-a6b9-10add14d0088\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " Nov 03 17:05:40.143506 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.143470 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-workload-certs\") pod \"c7ef9905-f189-4f71-a6b9-10add14d0088\" (UID: \"c7ef9905-f189-4f71-a6b9-10add14d0088\") " Nov 03 17:05:40.143728 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.143690 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-credential-socket" pod "c7ef9905-f189-4f71-a6b9-10add14d0088" (UID: "c7ef9905-f189-4f71-a6b9-10add14d0088"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:05:40.143836 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.143800 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-workload-certs" pod "c7ef9905-f189-4f71-a6b9-10add14d0088" (UID: "c7ef9905-f189-4f71-a6b9-10add14d0088"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:05:40.144609 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.144586 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:40.144609 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.144608 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:40.144832 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.144618 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:40.144832 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.144627 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ef9905-f189-4f71-a6b9-10add14d0088-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:40.144832 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.144634 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:40.145691 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.145666 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-podinfo" pod "c7ef9905-f189-4f71-a6b9-10add14d0088" (UID: "c7ef9905-f189-4f71-a6b9-10add14d0088"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:05:40.145812 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.145764 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-token" pod "c7ef9905-f189-4f71-a6b9-10add14d0088" (UID: "c7ef9905-f189-4f71-a6b9-10add14d0088"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:40.145945 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.145921 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-envoy" pod "c7ef9905-f189-4f71-a6b9-10add14d0088" (UID: "c7ef9905-f189-4f71-a6b9-10add14d0088"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:05:40.146037 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.145957 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ef9905-f189-4f71-a6b9-10add14d0088-kube-api-access-4dxtw" pod "c7ef9905-f189-4f71-a6b9-10add14d0088" (UID: "c7ef9905-f189-4f71-a6b9-10add14d0088"). InnerVolumeSpecName "kube-api-access-4dxtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:40.246594 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.246561 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:40.246594 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.246591 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:40.246594 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.246600 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dxtw\" (UniqueName: \"kubernetes.io/projected/c7ef9905-f189-4f71-a6b9-10add14d0088-kube-api-access-4dxtw\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:40.246933 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.246610 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ef9905-f189-4f71-a6b9-10add14d0088-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:40.586187 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.586156 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w"] Nov 03 17:05:40.725101 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.725063 4947 generic.go:356] "Generic (PLEG): container finished" podID="c7ef9905-f189-4f71-a6b9-10add14d0088" containerID="78c1d6a34d89dc9a909741647427717b9959c6cccfde0e8d7e16c1e9e9b3406f" exitCode=0 Nov 03 17:05:40.725392 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.725104 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" Nov 03 17:05:40.725392 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.725134 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" event={"ID":"c7ef9905-f189-4f71-a6b9-10add14d0088","Type":"ContainerDied","Data":"78c1d6a34d89dc9a909741647427717b9959c6cccfde0e8d7e16c1e9e9b3406f"} Nov 03 17:05:40.725392 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.725192 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n" event={"ID":"c7ef9905-f189-4f71-a6b9-10add14d0088","Type":"ContainerDied","Data":"564451b65d2b9cf07cd261cc8f6f947a61a0d4929970761fbea12cb897c71300"} Nov 03 17:05:40.725392 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.725240 4947 scope.go:122] "RemoveContainer" containerID="78c1d6a34d89dc9a909741647427717b9959c6cccfde0e8d7e16c1e9e9b3406f" Nov 03 17:05:40.736338 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.736318 4947 scope.go:122] "RemoveContainer" containerID="78c1d6a34d89dc9a909741647427717b9959c6cccfde0e8d7e16c1e9e9b3406f" Nov 03 17:05:40.736640 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:05:40.736614 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78c1d6a34d89dc9a909741647427717b9959c6cccfde0e8d7e16c1e9e9b3406f\": container with ID starting with 78c1d6a34d89dc9a909741647427717b9959c6cccfde0e8d7e16c1e9e9b3406f not found: ID does not exist" containerID="78c1d6a34d89dc9a909741647427717b9959c6cccfde0e8d7e16c1e9e9b3406f" Nov 03 17:05:40.736769 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.736648 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78c1d6a34d89dc9a909741647427717b9959c6cccfde0e8d7e16c1e9e9b3406f"} err="failed to get container status \"78c1d6a34d89dc9a909741647427717b9959c6cccfde0e8d7e16c1e9e9b3406f\": rpc error: code = NotFound desc = could not find container \"78c1d6a34d89dc9a909741647427717b9959c6cccfde0e8d7e16c1e9e9b3406f\": container with ID starting with 78c1d6a34d89dc9a909741647427717b9959c6cccfde0e8d7e16c1e9e9b3406f not found: ID does not exist" Nov 03 17:05:40.784668 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.784635 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n"] Nov 03 17:05:40.797292 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:40.797264 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-2b75n"] Nov 03 17:05:41.522332 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.522304 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 17:05:41.566264 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.566231 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-kube-api-access-dbq58\" (UniqueName: \"kubernetes.io/projected/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-kube-api-access-dbq58\") pod \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " Nov 03 17:05:41.566439 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.566382 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-podinfo\") pod \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " Nov 03 17:05:41.566489 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.566447 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istiod-ca-cert\") pod \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " Nov 03 17:05:41.566489 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.566483 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-credential-socket\") pod \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " Nov 03 17:05:41.566602 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.566507 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-envoy\") pod \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " Nov 03 17:05:41.566602 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.566565 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-workload-socket\") pod \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " Nov 03 17:05:41.567001 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.566969 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istiod-ca-cert" pod "d281375d-bb6c-4ac6-b2fb-8bd4cb476178" (UID: "d281375d-bb6c-4ac6-b2fb-8bd4cb476178"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:05:41.567113 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.567083 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-workload-socket" pod "d281375d-bb6c-4ac6-b2fb-8bd4cb476178" (UID: "d281375d-bb6c-4ac6-b2fb-8bd4cb476178"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:05:41.567174 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.567111 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-token\" (UniqueName: \"kubernetes.io/projected/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-token\") pod \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " Nov 03 17:05:41.567174 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.567149 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-credential-socket" pod "d281375d-bb6c-4ac6-b2fb-8bd4cb476178" (UID: "d281375d-bb6c-4ac6-b2fb-8bd4cb476178"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:05:41.567295 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.567177 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-workload-certs\") pod \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " Nov 03 17:05:41.567353 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.567291 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-data\") pod \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\" (UID: \"d281375d-bb6c-4ac6-b2fb-8bd4cb476178\") " Nov 03 17:05:41.567353 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.567339 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-workload-certs" pod "d281375d-bb6c-4ac6-b2fb-8bd4cb476178" (UID: "d281375d-bb6c-4ac6-b2fb-8bd4cb476178"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:05:41.567580 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.567549 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-data" pod "d281375d-bb6c-4ac6-b2fb-8bd4cb476178" (UID: "d281375d-bb6c-4ac6-b2fb-8bd4cb476178"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:05:41.568069 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.568045 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:41.568180 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.568076 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:41.568180 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.568090 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:41.568180 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.568103 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:41.568180 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.568115 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:41.569200 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.569176 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-kube-api-access-dbq58" pod "d281375d-bb6c-4ac6-b2fb-8bd4cb476178" (UID: "d281375d-bb6c-4ac6-b2fb-8bd4cb476178"). InnerVolumeSpecName "kube-api-access-dbq58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:41.569421 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.569398 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-envoy" pod "d281375d-bb6c-4ac6-b2fb-8bd4cb476178" (UID: "d281375d-bb6c-4ac6-b2fb-8bd4cb476178"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:05:41.569620 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.569601 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-podinfo" pod "d281375d-bb6c-4ac6-b2fb-8bd4cb476178" (UID: "d281375d-bb6c-4ac6-b2fb-8bd4cb476178"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:05:41.569763 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.569746 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-token" pod "d281375d-bb6c-4ac6-b2fb-8bd4cb476178" (UID: "d281375d-bb6c-4ac6-b2fb-8bd4cb476178"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:41.670543 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.670507 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:41.670543 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.670543 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbq58\" (UniqueName: \"kubernetes.io/projected/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-kube-api-access-dbq58\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:41.670889 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.670559 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:41.670889 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.670573 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d281375d-bb6c-4ac6-b2fb-8bd4cb476178-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:41.735896 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.735856 4947 generic.go:356] "Generic (PLEG): container finished" podID="d281375d-bb6c-4ac6-b2fb-8bd4cb476178" containerID="58d3051d9350dc8cc3494e1eeb3fc792c8029a29bcf41f3d3614d697e83ffd7e" exitCode=0 Nov 03 17:05:41.736061 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.735894 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" event={"ID":"d281375d-bb6c-4ac6-b2fb-8bd4cb476178","Type":"ContainerDied","Data":"58d3051d9350dc8cc3494e1eeb3fc792c8029a29bcf41f3d3614d697e83ffd7e"} Nov 03 17:05:41.736061 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.735960 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" event={"ID":"d281375d-bb6c-4ac6-b2fb-8bd4cb476178","Type":"ContainerDied","Data":"758978b20617dd13890de58bbdf059f98c2ac01bac9b50b7dd6e71cf0934008b"} Nov 03 17:05:41.736061 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.735905 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf" Nov 03 17:05:41.736242 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.735986 4947 scope.go:122] "RemoveContainer" containerID="58d3051d9350dc8cc3494e1eeb3fc792c8029a29bcf41f3d3614d697e83ffd7e" Nov 03 17:05:41.762299 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.762275 4947 scope.go:122] "RemoveContainer" containerID="58d3051d9350dc8cc3494e1eeb3fc792c8029a29bcf41f3d3614d697e83ffd7e" Nov 03 17:05:41.762616 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:05:41.762595 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d3051d9350dc8cc3494e1eeb3fc792c8029a29bcf41f3d3614d697e83ffd7e\": container with ID starting with 58d3051d9350dc8cc3494e1eeb3fc792c8029a29bcf41f3d3614d697e83ffd7e not found: ID does not exist" containerID="58d3051d9350dc8cc3494e1eeb3fc792c8029a29bcf41f3d3614d697e83ffd7e" Nov 03 17:05:41.762676 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.762625 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d3051d9350dc8cc3494e1eeb3fc792c8029a29bcf41f3d3614d697e83ffd7e"} err="failed to get container status \"58d3051d9350dc8cc3494e1eeb3fc792c8029a29bcf41f3d3614d697e83ffd7e\": rpc error: code = NotFound desc = could not find container \"58d3051d9350dc8cc3494e1eeb3fc792c8029a29bcf41f3d3614d697e83ffd7e\": container with ID starting with 58d3051d9350dc8cc3494e1eeb3fc792c8029a29bcf41f3d3614d697e83ffd7e not found: ID does not exist" Nov 03 17:05:41.847558 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.847524 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf"] Nov 03 17:05:41.855555 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:41.855223 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-kftbf"] Nov 03 17:05:42.569343 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:42.569312 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ef9905-f189-4f71-a6b9-10add14d0088" path="/var/lib/kubelet/pods/c7ef9905-f189-4f71-a6b9-10add14d0088/volumes" Nov 03 17:05:42.569914 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:42.569895 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d281375d-bb6c-4ac6-b2fb-8bd4cb476178" path="/var/lib/kubelet/pods/d281375d-bb6c-4ac6-b2fb-8bd4cb476178/volumes" Nov 03 17:05:45.586727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:45.586651 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" podUID="bc89793f-a6fb-4546-a4bf-948a747e66fa" containerName="payload-processing" containerID="cri-o://40d2ef5fecaf82e8691c1fa5530d30d61704b6c5f4bfe8434797bb26f62286d8" gracePeriod=25 Nov 03 17:05:45.888674 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:45.888594 4947 generic.go:356] "Generic (PLEG): container finished" podID="bc89793f-a6fb-4546-a4bf-948a747e66fa" containerID="40d2ef5fecaf82e8691c1fa5530d30d61704b6c5f4bfe8434797bb26f62286d8" exitCode=0 Nov 03 17:05:45.888868 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:45.888736 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" event={"ID":"bc89793f-a6fb-4546-a4bf-948a747e66fa","Type":"ContainerDied","Data":"40d2ef5fecaf82e8691c1fa5530d30d61704b6c5f4bfe8434797bb26f62286d8"} Nov 03 17:05:46.267964 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:46.267928 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" Nov 03 17:05:46.295411 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:46.295378 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc89793f-a6fb-4546-a4bf-948a747e66fa-kube-api-access-lx6z4\" (UniqueName: \"kubernetes.io/projected/bc89793f-a6fb-4546-a4bf-948a747e66fa-kube-api-access-lx6z4\") pod \"bc89793f-a6fb-4546-a4bf-948a747e66fa\" (UID: \"bc89793f-a6fb-4546-a4bf-948a747e66fa\") " Nov 03 17:05:46.295625 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:46.295449 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc89793f-a6fb-4546-a4bf-948a747e66fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc89793f-a6fb-4546-a4bf-948a747e66fa-plugins-config-volume\") pod \"bc89793f-a6fb-4546-a4bf-948a747e66fa\" (UID: \"bc89793f-a6fb-4546-a4bf-948a747e66fa\") " Nov 03 17:05:46.295625 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:46.295479 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc89793f-a6fb-4546-a4bf-948a747e66fa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc89793f-a6fb-4546-a4bf-948a747e66fa-routing-overlay-volume\") pod \"bc89793f-a6fb-4546-a4bf-948a747e66fa\" (UID: \"bc89793f-a6fb-4546-a4bf-948a747e66fa\") " Nov 03 17:05:46.295797 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:46.295658 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc89793f-a6fb-4546-a4bf-948a747e66fa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc89793f-a6fb-4546-a4bf-948a747e66fa-provider-credentials\") pod \"bc89793f-a6fb-4546-a4bf-948a747e66fa\" (UID: \"bc89793f-a6fb-4546-a4bf-948a747e66fa\") " Nov 03 17:05:46.295913 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:46.295892 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc89793f-a6fb-4546-a4bf-948a747e66fa-routing-overlay-volume" pod "bc89793f-a6fb-4546-a4bf-948a747e66fa" (UID: "bc89793f-a6fb-4546-a4bf-948a747e66fa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:05:46.295990 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:46.295906 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc89793f-a6fb-4546-a4bf-948a747e66fa-plugins-config-volume" pod "bc89793f-a6fb-4546-a4bf-948a747e66fa" (UID: "bc89793f-a6fb-4546-a4bf-948a747e66fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:05:46.296295 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:46.296274 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc89793f-a6fb-4546-a4bf-948a747e66fa-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:46.296364 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:46.296300 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc89793f-a6fb-4546-a4bf-948a747e66fa-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:46.298053 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:46.298019 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc89793f-a6fb-4546-a4bf-948a747e66fa-provider-credentials" pod "bc89793f-a6fb-4546-a4bf-948a747e66fa" (UID: "bc89793f-a6fb-4546-a4bf-948a747e66fa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:46.298182 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:46.298083 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc89793f-a6fb-4546-a4bf-948a747e66fa-kube-api-access-lx6z4" pod "bc89793f-a6fb-4546-a4bf-948a747e66fa" (UID: "bc89793f-a6fb-4546-a4bf-948a747e66fa"). InnerVolumeSpecName "kube-api-access-lx6z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:05:46.397678 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:46.397641 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc89793f-a6fb-4546-a4bf-948a747e66fa-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:46.397678 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:46.397671 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx6z4\" (UniqueName: \"kubernetes.io/projected/bc89793f-a6fb-4546-a4bf-948a747e66fa-kube-api-access-lx6z4\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:05:46.987771 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:46.986284 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" event={"ID":"bc89793f-a6fb-4546-a4bf-948a747e66fa","Type":"ContainerDied","Data":"ae18559cc8b1a1f327da9c48b88e14c891c07fdf29803b80b19a3418612dc384"} Nov 03 17:05:46.987771 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:46.986365 4947 scope.go:122] "RemoveContainer" containerID="40d2ef5fecaf82e8691c1fa5530d30d61704b6c5f4bfe8434797bb26f62286d8" Nov 03 17:05:46.987771 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:46.986542 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w" Nov 03 17:05:47.090311 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.090278 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w"] Nov 03 17:05:47.099381 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.099351 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c4p4z7w"] Nov 03 17:05:47.102982 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.102960 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s"] Nov 03 17:05:47.103874 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.103857 4947 cpu_manager.go:414] "containerMap: removing container" podUID="bc89793f-a6fb-4546-a4bf-948a747e66fa" containerName="payload-processing" Nov 03 17:05:47.103947 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.103876 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc89793f-a6fb-4546-a4bf-948a747e66fa" containerName="payload-processing" Nov 03 17:05:47.103947 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.103899 4947 cpu_manager.go:414] "containerMap: removing container" podUID="c7ef9905-f189-4f71-a6b9-10add14d0088" containerName="istio-proxy" Nov 03 17:05:47.103947 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.103904 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ef9905-f189-4f71-a6b9-10add14d0088" containerName="istio-proxy" Nov 03 17:05:47.103947 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.103940 4947 cpu_manager.go:414] "containerMap: removing container" podUID="d281375d-bb6c-4ac6-b2fb-8bd4cb476178" containerName="istio-proxy" Nov 03 17:05:47.103947 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.103946 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d281375d-bb6c-4ac6-b2fb-8bd4cb476178" containerName="istio-proxy" Nov 03 17:05:47.104118 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.103956 4947 cpu_manager.go:414] "containerMap: removing container" podUID="e2733a9d-062f-4ea5-927b-97d974df3f92" containerName="payload-processing" Nov 03 17:05:47.104118 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.103961 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2733a9d-062f-4ea5-927b-97d974df3f92" containerName="payload-processing" Nov 03 17:05:47.104118 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.103972 4947 cpu_manager.go:414] "containerMap: removing container" podUID="e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18" containerName="payload-processing" Nov 03 17:05:47.104118 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.103978 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18" containerName="payload-processing" Nov 03 17:05:47.104118 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.104079 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="d281375d-bb6c-4ac6-b2fb-8bd4cb476178" containerName="istio-proxy" Nov 03 17:05:47.104118 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.104087 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2733a9d-062f-4ea5-927b-97d974df3f92" containerName="payload-processing" Nov 03 17:05:47.104118 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.104096 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2ae9bb6-1d56-4007-afc6-9a4b9c4bcb18" containerName="payload-processing" Nov 03 17:05:47.104118 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.104104 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc89793f-a6fb-4546-a4bf-948a747e66fa" containerName="payload-processing" Nov 03 17:05:47.104118 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.104112 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ef9905-f189-4f71-a6b9-10add14d0088" containerName="istio-proxy" Nov 03 17:05:47.109401 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.109379 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" Nov 03 17:05:47.127214 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.127183 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1f83c0\"/\"e2e-cmp-legacy-2c3746-provider-cred\"" Nov 03 17:05:47.127655 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.127586 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1f83c0\"/\"payload-processing-external-model-e2e--91a4f230-dockercfg-t8g8q\"" Nov 03 17:05:47.129463 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.129443 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1f83c0\"/\"routing-overlay\"" Nov 03 17:05:47.129718 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.129476 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1f83c0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-1f83c0\"" Nov 03 17:05:47.129718 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.129533 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1f83c0\"/\"kube-root-ca.crt\"" Nov 03 17:05:47.129718 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.129528 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1f83c0\"/\"openshift-service-ca.crt\"" Nov 03 17:05:47.131950 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.131930 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s"] Nov 03 17:05:47.226913 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.226880 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7818056-d43b-4560-9e3f-ad309fd0dd9d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s\" (UID: \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" Nov 03 17:05:47.227186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.227155 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tlh2\" (UniqueName: \"kubernetes.io/projected/c7818056-d43b-4560-9e3f-ad309fd0dd9d-kube-api-access-7tlh2\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s\" (UID: \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" Nov 03 17:05:47.227304 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.227286 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c7818056-d43b-4560-9e3f-ad309fd0dd9d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s\" (UID: \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" Nov 03 17:05:47.227386 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.227317 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c7818056-d43b-4560-9e3f-ad309fd0dd9d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s\" (UID: \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" Nov 03 17:05:47.331816 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.329860 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c7818056-d43b-4560-9e3f-ad309fd0dd9d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s\" (UID: \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" Nov 03 17:05:47.335830 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.335794 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c7818056-d43b-4560-9e3f-ad309fd0dd9d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s\" (UID: \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" Nov 03 17:05:47.353720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.333096 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c7818056-d43b-4560-9e3f-ad309fd0dd9d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s\" (UID: \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" Nov 03 17:05:47.354087 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.354058 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7818056-d43b-4560-9e3f-ad309fd0dd9d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s\" (UID: \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" Nov 03 17:05:47.354495 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.354470 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tlh2\" (UniqueName: \"kubernetes.io/projected/c7818056-d43b-4560-9e3f-ad309fd0dd9d-kube-api-access-7tlh2\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s\" (UID: \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" Nov 03 17:05:47.369982 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.355483 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7818056-d43b-4560-9e3f-ad309fd0dd9d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s\" (UID: \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" Nov 03 17:05:47.386359 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.386324 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c7818056-d43b-4560-9e3f-ad309fd0dd9d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s\" (UID: \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" Nov 03 17:05:47.427773 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.426863 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tlh2\" (UniqueName: \"kubernetes.io/projected/c7818056-d43b-4560-9e3f-ad309fd0dd9d-kube-api-access-7tlh2\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s\" (UID: \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" Nov 03 17:05:47.725644 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:47.725562 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" Nov 03 17:05:48.069188 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:05:48.069153 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7818056_d43b_4560_9e3f_ad309fd0dd9d.slice/crio-f516959586ea029dbec641f2c71cc93b2dffd783603b608bd6233d1afe209a6c WatchSource:0}: Error finding container f516959586ea029dbec641f2c71cc93b2dffd783603b608bd6233d1afe209a6c: Status 404 returned error can't find the container with id f516959586ea029dbec641f2c71cc93b2dffd783603b608bd6233d1afe209a6c Nov 03 17:05:48.070418 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:48.070391 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s"] Nov 03 17:05:48.569383 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:48.569352 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc89793f-a6fb-4546-a4bf-948a747e66fa" path="/var/lib/kubelet/pods/bc89793f-a6fb-4546-a4bf-948a747e66fa/volumes" Nov 03 17:05:49.003947 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:49.003908 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" event={"ID":"c7818056-d43b-4560-9e3f-ad309fd0dd9d","Type":"ContainerStarted","Data":"750da36bdc2c4f089b7580f6ac2f29f462317b55e3d7a575a46d95485b49da04"} Nov 03 17:05:49.004152 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:49.003970 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" event={"ID":"c7818056-d43b-4560-9e3f-ad309fd0dd9d","Type":"ContainerStarted","Data":"f516959586ea029dbec641f2c71cc93b2dffd783603b608bd6233d1afe209a6c"} Nov 03 17:05:49.004152 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:49.004047 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" Nov 03 17:05:49.051037 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:05:49.050976 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" podStartSLOduration=2.050960624 podStartE2EDuration="2.050960624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:05:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:05:49.048519182 +0000 UTC m=+3092.637519153" watchObservedRunningTime="2026-11-03 17:05:49.050960624 +0000 UTC m=+3092.639960593" Nov 03 17:06:00.016516 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:00.016423 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" Nov 03 17:06:00.136377 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:00.135529 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:06:19.080727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:19.078959 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg"] Nov 03 17:06:19.087742 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:19.087354 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" Nov 03 17:06:19.107410 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:19.107378 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg"] Nov 03 17:06:19.152380 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:19.152338 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvd9g\" (UniqueName: \"kubernetes.io/projected/4ac31687-2522-4e91-b686-36f48e7f3d15-kube-api-access-bvd9g\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg\" (UID: \"4ac31687-2522-4e91-b686-36f48e7f3d15\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" Nov 03 17:06:19.152588 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:19.152520 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ac31687-2522-4e91-b686-36f48e7f3d15-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg\" (UID: \"4ac31687-2522-4e91-b686-36f48e7f3d15\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" Nov 03 17:06:19.152670 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:19.152638 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ac31687-2522-4e91-b686-36f48e7f3d15-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg\" (UID: \"4ac31687-2522-4e91-b686-36f48e7f3d15\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" Nov 03 17:06:19.152751 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:19.152714 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac31687-2522-4e91-b686-36f48e7f3d15-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg\" (UID: \"4ac31687-2522-4e91-b686-36f48e7f3d15\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" Nov 03 17:06:19.254784 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:19.254691 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ac31687-2522-4e91-b686-36f48e7f3d15-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg\" (UID: \"4ac31687-2522-4e91-b686-36f48e7f3d15\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" Nov 03 17:06:19.254969 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:19.254810 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac31687-2522-4e91-b686-36f48e7f3d15-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg\" (UID: \"4ac31687-2522-4e91-b686-36f48e7f3d15\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" Nov 03 17:06:19.255013 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:19.254993 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvd9g\" (UniqueName: \"kubernetes.io/projected/4ac31687-2522-4e91-b686-36f48e7f3d15-kube-api-access-bvd9g\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg\" (UID: \"4ac31687-2522-4e91-b686-36f48e7f3d15\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" Nov 03 17:06:19.255087 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:19.255071 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ac31687-2522-4e91-b686-36f48e7f3d15-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg\" (UID: \"4ac31687-2522-4e91-b686-36f48e7f3d15\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" Nov 03 17:06:19.255779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:19.255756 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac31687-2522-4e91-b686-36f48e7f3d15-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg\" (UID: \"4ac31687-2522-4e91-b686-36f48e7f3d15\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" Nov 03 17:06:19.255925 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:19.255848 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ac31687-2522-4e91-b686-36f48e7f3d15-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg\" (UID: \"4ac31687-2522-4e91-b686-36f48e7f3d15\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" Nov 03 17:06:19.257628 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:19.257605 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ac31687-2522-4e91-b686-36f48e7f3d15-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg\" (UID: \"4ac31687-2522-4e91-b686-36f48e7f3d15\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" Nov 03 17:06:19.280150 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:19.280116 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvd9g\" (UniqueName: \"kubernetes.io/projected/4ac31687-2522-4e91-b686-36f48e7f3d15-kube-api-access-bvd9g\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg\" (UID: \"4ac31687-2522-4e91-b686-36f48e7f3d15\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" Nov 03 17:06:19.443394 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:19.443311 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" Nov 03 17:06:19.761814 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:06:19.761778 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac31687_2522_4e91_b686_36f48e7f3d15.slice/crio-7065ae0aa680adc2478ca32a0c03533b627392c134f894273e2170f1029c92f2 WatchSource:0}: Error finding container 7065ae0aa680adc2478ca32a0c03533b627392c134f894273e2170f1029c92f2: Status 404 returned error can't find the container with id 7065ae0aa680adc2478ca32a0c03533b627392c134f894273e2170f1029c92f2 Nov 03 17:06:19.764598 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:19.763337 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg"] Nov 03 17:06:20.284894 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:20.284858 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" event={"ID":"4ac31687-2522-4e91-b686-36f48e7f3d15","Type":"ContainerStarted","Data":"281162ec2b121b6f6c741d11ea88cf2cbb284fa2bcfde896fb4f9951a3640770"} Nov 03 17:06:20.285326 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:20.284916 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" event={"ID":"4ac31687-2522-4e91-b686-36f48e7f3d15","Type":"ContainerStarted","Data":"7065ae0aa680adc2478ca32a0c03533b627392c134f894273e2170f1029c92f2"} Nov 03 17:06:20.285326 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:20.284961 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" Nov 03 17:06:20.333530 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:20.333472 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" podStartSLOduration=1.333456071 podStartE2EDuration="1.333456071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:06:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:06:20.329325935 +0000 UTC m=+3123.918325904" watchObservedRunningTime="2026-11-03 17:06:20.333456071 +0000 UTC m=+3123.922456037" Nov 03 17:06:22.936106 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:22.936068 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c"] Nov 03 17:06:22.936667 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:22.936360 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" podUID="05078fe5-de88-4050-829f-d2a51f0328a8" containerName="istio-proxy" containerID="cri-o://25eee918399d70e921a6a8551df2979b8c872cc32eadea09dadca047825cecc7" gracePeriod=30 Nov 03 17:06:28.212661 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.212586 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:06:28.310847 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.310807 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-credential-socket\") pod \"05078fe5-de88-4050-829f-d2a51f0328a8\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " Nov 03 17:06:28.311023 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.310932 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-istio-data\") pod \"05078fe5-de88-4050-829f-d2a51f0328a8\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " Nov 03 17:06:28.311023 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.310968 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-istio-envoy\") pod \"05078fe5-de88-4050-829f-d2a51f0328a8\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " Nov 03 17:06:28.311169 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.311058 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-workload-socket\") pod \"05078fe5-de88-4050-829f-d2a51f0328a8\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " Nov 03 17:06:28.311169 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.311135 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05078fe5-de88-4050-829f-d2a51f0328a8-kube-api-access-cs2pt\" (UniqueName: \"kubernetes.io/projected/05078fe5-de88-4050-829f-d2a51f0328a8-kube-api-access-cs2pt\") pod \"05078fe5-de88-4050-829f-d2a51f0328a8\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " Nov 03 17:06:28.311282 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.311185 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-credential-socket" pod "05078fe5-de88-4050-829f-d2a51f0328a8" (UID: "05078fe5-de88-4050-829f-d2a51f0328a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:06:28.311282 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.311233 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-istio-data" pod "05078fe5-de88-4050-829f-d2a51f0328a8" (UID: "05078fe5-de88-4050-829f-d2a51f0328a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:06:28.311282 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.311263 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-workload-certs\") pod \"05078fe5-de88-4050-829f-d2a51f0328a8\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " Nov 03 17:06:28.311436 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.311335 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05078fe5-de88-4050-829f-d2a51f0328a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05078fe5-de88-4050-829f-d2a51f0328a8-istio-podinfo\") pod \"05078fe5-de88-4050-829f-d2a51f0328a8\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " Nov 03 17:06:28.311489 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.311463 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05078fe5-de88-4050-829f-d2a51f0328a8-istio-token\" (UniqueName: \"kubernetes.io/projected/05078fe5-de88-4050-829f-d2a51f0328a8-istio-token\") pod \"05078fe5-de88-4050-829f-d2a51f0328a8\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " Nov 03 17:06:28.311550 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.311492 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-workload-socket" pod "05078fe5-de88-4050-829f-d2a51f0328a8" (UID: "05078fe5-de88-4050-829f-d2a51f0328a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:06:28.311550 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.311529 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05078fe5-de88-4050-829f-d2a51f0328a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05078fe5-de88-4050-829f-d2a51f0328a8-istiod-ca-cert\") pod \"05078fe5-de88-4050-829f-d2a51f0328a8\" (UID: \"05078fe5-de88-4050-829f-d2a51f0328a8\") " Nov 03 17:06:28.311683 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.311664 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-workload-certs" pod "05078fe5-de88-4050-829f-d2a51f0328a8" (UID: "05078fe5-de88-4050-829f-d2a51f0328a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:06:28.312176 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.312156 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:06:28.312325 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.312313 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:06:28.312411 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.312401 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:06:28.312489 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.312479 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:06:28.312925 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.312907 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05078fe5-de88-4050-829f-d2a51f0328a8-istiod-ca-cert" pod "05078fe5-de88-4050-829f-d2a51f0328a8" (UID: "05078fe5-de88-4050-829f-d2a51f0328a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:06:28.314549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.314518 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05078fe5-de88-4050-829f-d2a51f0328a8-istio-podinfo" pod "05078fe5-de88-4050-829f-d2a51f0328a8" (UID: "05078fe5-de88-4050-829f-d2a51f0328a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:06:28.314661 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.314578 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05078fe5-de88-4050-829f-d2a51f0328a8-kube-api-access-cs2pt" pod "05078fe5-de88-4050-829f-d2a51f0328a8" (UID: "05078fe5-de88-4050-829f-d2a51f0328a8"). InnerVolumeSpecName "kube-api-access-cs2pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:06:28.314850 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.314659 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-istio-envoy" pod "05078fe5-de88-4050-829f-d2a51f0328a8" (UID: "05078fe5-de88-4050-829f-d2a51f0328a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:06:28.315870 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.315845 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05078fe5-de88-4050-829f-d2a51f0328a8-istio-token" pod "05078fe5-de88-4050-829f-d2a51f0328a8" (UID: "05078fe5-de88-4050-829f-d2a51f0328a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:06:28.359167 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.359131 4947 generic.go:356] "Generic (PLEG): container finished" podID="05078fe5-de88-4050-829f-d2a51f0328a8" containerID="25eee918399d70e921a6a8551df2979b8c872cc32eadea09dadca047825cecc7" exitCode=0 Nov 03 17:06:28.359328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.359193 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" event={"ID":"05078fe5-de88-4050-829f-d2a51f0328a8","Type":"ContainerDied","Data":"25eee918399d70e921a6a8551df2979b8c872cc32eadea09dadca047825cecc7"} Nov 03 17:06:28.359328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.359198 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" Nov 03 17:06:28.359328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.359276 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c" event={"ID":"05078fe5-de88-4050-829f-d2a51f0328a8","Type":"ContainerDied","Data":"204e9b42e466480ce412f3f04ae913678b0c77030a67bb4d3f755d1147b7406b"} Nov 03 17:06:28.359328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.359305 4947 scope.go:122] "RemoveContainer" containerID="25eee918399d70e921a6a8551df2979b8c872cc32eadea09dadca047825cecc7" Nov 03 17:06:28.373150 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.373123 4947 scope.go:122] "RemoveContainer" containerID="25eee918399d70e921a6a8551df2979b8c872cc32eadea09dadca047825cecc7" Nov 03 17:06:28.373447 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:28.373425 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25eee918399d70e921a6a8551df2979b8c872cc32eadea09dadca047825cecc7\": container with ID starting with 25eee918399d70e921a6a8551df2979b8c872cc32eadea09dadca047825cecc7 not found: ID does not exist" containerID="25eee918399d70e921a6a8551df2979b8c872cc32eadea09dadca047825cecc7" Nov 03 17:06:28.373501 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.373460 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25eee918399d70e921a6a8551df2979b8c872cc32eadea09dadca047825cecc7"} err="failed to get container status \"25eee918399d70e921a6a8551df2979b8c872cc32eadea09dadca047825cecc7\": rpc error: code = NotFound desc = could not find container \"25eee918399d70e921a6a8551df2979b8c872cc32eadea09dadca047825cecc7\": container with ID starting with 25eee918399d70e921a6a8551df2979b8c872cc32eadea09dadca047825cecc7 not found: ID does not exist" Nov 03 17:06:28.414362 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.414326 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs2pt\" (UniqueName: \"kubernetes.io/projected/05078fe5-de88-4050-829f-d2a51f0328a8-kube-api-access-cs2pt\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:06:28.414362 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.414358 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05078fe5-de88-4050-829f-d2a51f0328a8-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:06:28.414362 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.414369 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05078fe5-de88-4050-829f-d2a51f0328a8-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:06:28.414620 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.414377 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05078fe5-de88-4050-829f-d2a51f0328a8-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:06:28.414620 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.414385 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05078fe5-de88-4050-829f-d2a51f0328a8-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:06:28.418191 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.418162 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c"] Nov 03 17:06:28.423760 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.423735 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-j4w8c"] Nov 03 17:06:28.568415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:28.568385 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05078fe5-de88-4050-829f-d2a51f0328a8" path="/var/lib/kubelet/pods/05078fe5-de88-4050-829f-d2a51f0328a8/volumes" Nov 03 17:06:30.130919 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:30.130879 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:06:31.198687 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.198647 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:06:31.259294 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.259251 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg"] Nov 03 17:06:31.264924 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.264888 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" Nov 03 17:06:31.288305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.288267 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk"] Nov 03 17:06:31.289103 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.289079 4947 cpu_manager.go:414] "containerMap: removing container" podUID="05078fe5-de88-4050-829f-d2a51f0328a8" containerName="istio-proxy" Nov 03 17:06:31.289103 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.289098 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05078fe5-de88-4050-829f-d2a51f0328a8" containerName="istio-proxy" Nov 03 17:06:31.289297 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.289232 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="05078fe5-de88-4050-829f-d2a51f0328a8" containerName="istio-proxy" Nov 03 17:06:31.294605 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.294583 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" Nov 03 17:06:31.319415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.319376 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk"] Nov 03 17:06:31.415890 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.415855 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf4bg\" (UniqueName: \"kubernetes.io/projected/230f71b3-37f5-4ca2-a616-c139cf9b6c93-kube-api-access-mf4bg\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk\" (UID: \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" Nov 03 17:06:31.416095 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.415906 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/230f71b3-37f5-4ca2-a616-c139cf9b6c93-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk\" (UID: \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" Nov 03 17:06:31.416095 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.415936 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230f71b3-37f5-4ca2-a616-c139cf9b6c93-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk\" (UID: \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" Nov 03 17:06:31.416095 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.416000 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/230f71b3-37f5-4ca2-a616-c139cf9b6c93-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk\" (UID: \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" Nov 03 17:06:31.526906 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.526859 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf4bg\" (UniqueName: \"kubernetes.io/projected/230f71b3-37f5-4ca2-a616-c139cf9b6c93-kube-api-access-mf4bg\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk\" (UID: \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" Nov 03 17:06:31.529209 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.529179 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/230f71b3-37f5-4ca2-a616-c139cf9b6c93-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk\" (UID: \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" Nov 03 17:06:31.530930 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.530906 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230f71b3-37f5-4ca2-a616-c139cf9b6c93-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk\" (UID: \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" Nov 03 17:06:31.532162 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.532140 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/230f71b3-37f5-4ca2-a616-c139cf9b6c93-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk\" (UID: \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" Nov 03 17:06:31.533236 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.532072 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230f71b3-37f5-4ca2-a616-c139cf9b6c93-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk\" (UID: \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" Nov 03 17:06:31.533568 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.530830 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/230f71b3-37f5-4ca2-a616-c139cf9b6c93-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk\" (UID: \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" Nov 03 17:06:31.535558 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.535535 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/230f71b3-37f5-4ca2-a616-c139cf9b6c93-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk\" (UID: \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" Nov 03 17:06:31.552939 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.552906 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf4bg\" (UniqueName: \"kubernetes.io/projected/230f71b3-37f5-4ca2-a616-c139cf9b6c93-kube-api-access-mf4bg\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk\" (UID: \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" Nov 03 17:06:31.609602 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.609559 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" Nov 03 17:06:31.923233 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:06:31.923203 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230f71b3_37f5_4ca2_a616_c139cf9b6c93.slice/crio-6856ea6029558e166f51bd3144ec9f00791805a2208c1fae0645d3701e4ecc46 WatchSource:0}: Error finding container 6856ea6029558e166f51bd3144ec9f00791805a2208c1fae0645d3701e4ecc46: Status 404 returned error can't find the container with id 6856ea6029558e166f51bd3144ec9f00791805a2208c1fae0645d3701e4ecc46 Nov 03 17:06:31.924862 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:31.924833 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk"] Nov 03 17:06:32.392983 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:32.392936 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" event={"ID":"230f71b3-37f5-4ca2-a616-c139cf9b6c93","Type":"ContainerStarted","Data":"a026ac2abc54bf841a9abd9a7c757a44f8aa9276dccea40ac69cda954e202adf"} Nov 03 17:06:32.393443 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:32.393000 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" event={"ID":"230f71b3-37f5-4ca2-a616-c139cf9b6c93","Type":"ContainerStarted","Data":"6856ea6029558e166f51bd3144ec9f00791805a2208c1fae0645d3701e4ecc46"} Nov 03 17:06:32.393443 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:32.393192 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" Nov 03 17:06:32.440467 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:32.440408 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" podStartSLOduration=1.4403903279999999 podStartE2EDuration="1.440390328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:06:32.437542886 +0000 UTC m=+3136.026542855" watchObservedRunningTime="2026-11-03 17:06:32.440390328 +0000 UTC m=+3136.029390309" Nov 03 17:06:36.260402 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:36.260349 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" podUID="4ac31687-2522-4e91-b686-36f48e7f3d15" containerName="payload-processing" containerID="cri-o://281162ec2b121b6f6c741d11ea88cf2cbb284fa2bcfde896fb4f9951a3640770" gracePeriod=25 Nov 03 17:06:36.447303 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:36.447234 4947 generic.go:356] "Generic (PLEG): container finished" podID="4ac31687-2522-4e91-b686-36f48e7f3d15" containerID="281162ec2b121b6f6c741d11ea88cf2cbb284fa2bcfde896fb4f9951a3640770" exitCode=0 Nov 03 17:06:36.449936 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:36.447488 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" event={"ID":"4ac31687-2522-4e91-b686-36f48e7f3d15","Type":"ContainerDied","Data":"281162ec2b121b6f6c741d11ea88cf2cbb284fa2bcfde896fb4f9951a3640770"} Nov 03 17:06:36.614267 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:36.614235 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" Nov 03 17:06:36.720595 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:36.720554 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac31687-2522-4e91-b686-36f48e7f3d15-kube-api-access-bvd9g\" (UniqueName: \"kubernetes.io/projected/4ac31687-2522-4e91-b686-36f48e7f3d15-kube-api-access-bvd9g\") pod \"4ac31687-2522-4e91-b686-36f48e7f3d15\" (UID: \"4ac31687-2522-4e91-b686-36f48e7f3d15\") " Nov 03 17:06:36.720840 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:36.720640 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ac31687-2522-4e91-b686-36f48e7f3d15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac31687-2522-4e91-b686-36f48e7f3d15-plugins-config-volume\") pod \"4ac31687-2522-4e91-b686-36f48e7f3d15\" (UID: \"4ac31687-2522-4e91-b686-36f48e7f3d15\") " Nov 03 17:06:36.720916 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:36.720847 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ac31687-2522-4e91-b686-36f48e7f3d15-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ac31687-2522-4e91-b686-36f48e7f3d15-routing-overlay-volume\") pod \"4ac31687-2522-4e91-b686-36f48e7f3d15\" (UID: \"4ac31687-2522-4e91-b686-36f48e7f3d15\") " Nov 03 17:06:36.720986 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:36.720972 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ac31687-2522-4e91-b686-36f48e7f3d15-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ac31687-2522-4e91-b686-36f48e7f3d15-provider-credentials\") pod \"4ac31687-2522-4e91-b686-36f48e7f3d15\" (UID: \"4ac31687-2522-4e91-b686-36f48e7f3d15\") " Nov 03 17:06:36.721201 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:36.721171 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac31687-2522-4e91-b686-36f48e7f3d15-plugins-config-volume" pod "4ac31687-2522-4e91-b686-36f48e7f3d15" (UID: "4ac31687-2522-4e91-b686-36f48e7f3d15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:06:36.721296 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:36.721269 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac31687-2522-4e91-b686-36f48e7f3d15-routing-overlay-volume" pod "4ac31687-2522-4e91-b686-36f48e7f3d15" (UID: "4ac31687-2522-4e91-b686-36f48e7f3d15"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:06:36.721532 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:36.721510 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6046c2/agc-cred-b833a8f9cdcf: secret "agc-cred-b833a8f9cdcf" not found Nov 03 17:06:36.721532 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:36.721530 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6046c2/agc-cred-f09ac2317b26: secret "agc-cred-f09ac2317b26" not found Nov 03 17:06:36.721733 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:36.721541 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv: [secret "agc-cred-b833a8f9cdcf" not found, secret "agc-cred-f09ac2317b26" not found] Nov 03 17:06:36.721733 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:36.721612 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-provider-credentials podName:49219a67-6c62-444a-aaff-4501ec769c84 nodeName:}" failed. No retries permitted until 2026-11-03 17:06:37.221577518 +0000 UTC m=+3140.810577465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" (UID: "49219a67-6c62-444a-aaff-4501ec769c84") : [secret "agc-cred-b833a8f9cdcf" not found, secret "agc-cred-f09ac2317b26" not found] Nov 03 17:06:36.722210 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:36.722164 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ac31687-2522-4e91-b686-36f48e7f3d15-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:06:36.722210 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:36.722196 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ac31687-2522-4e91-b686-36f48e7f3d15-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:06:36.723652 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:36.723620 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac31687-2522-4e91-b686-36f48e7f3d15-provider-credentials" pod "4ac31687-2522-4e91-b686-36f48e7f3d15" (UID: "4ac31687-2522-4e91-b686-36f48e7f3d15"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:06:36.723796 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:36.723759 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac31687-2522-4e91-b686-36f48e7f3d15-kube-api-access-bvd9g" pod "4ac31687-2522-4e91-b686-36f48e7f3d15" (UID: "4ac31687-2522-4e91-b686-36f48e7f3d15"). InnerVolumeSpecName "kube-api-access-bvd9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:06:36.823514 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:36.823424 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ac31687-2522-4e91-b686-36f48e7f3d15-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:06:36.823514 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:36.823456 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvd9g\" (UniqueName: \"kubernetes.io/projected/4ac31687-2522-4e91-b686-36f48e7f3d15-kube-api-access-bvd9g\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:06:37.229508 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:37.229433 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6046c2/agc-cred-b833a8f9cdcf: secret "agc-cred-b833a8f9cdcf" not found Nov 03 17:06:37.229508 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:37.229461 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6046c2/agc-cred-f09ac2317b26: secret "agc-cred-f09ac2317b26" not found Nov 03 17:06:37.229508 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:37.229470 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv: [secret "agc-cred-b833a8f9cdcf" not found, secret "agc-cred-f09ac2317b26" not found] Nov 03 17:06:37.229796 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:37.229553 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-provider-credentials podName:49219a67-6c62-444a-aaff-4501ec769c84 nodeName:}" failed. No retries permitted until 2026-11-03 17:06:38.229536604 +0000 UTC m=+3141.818536564 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" (UID: "49219a67-6c62-444a-aaff-4501ec769c84") : [secret "agc-cred-b833a8f9cdcf" not found, secret "agc-cred-f09ac2317b26" not found] Nov 03 17:06:37.456513 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:37.456484 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" Nov 03 17:06:37.456513 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:37.456493 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg" event={"ID":"4ac31687-2522-4e91-b686-36f48e7f3d15","Type":"ContainerDied","Data":"7065ae0aa680adc2478ca32a0c03533b627392c134f894273e2170f1029c92f2"} Nov 03 17:06:37.457072 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:37.456584 4947 scope.go:122] "RemoveContainer" containerID="281162ec2b121b6f6c741d11ea88cf2cbb284fa2bcfde896fb4f9951a3640770" Nov 03 17:06:37.514159 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:37.514126 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg"] Nov 03 17:06:37.526888 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:37.526859 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-5c7fclrqg"] Nov 03 17:06:38.245884 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:38.245845 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6046c2/agc-cred-b833a8f9cdcf: secret "agc-cred-b833a8f9cdcf" not found Nov 03 17:06:38.245884 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:38.245876 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6046c2/agc-cred-f09ac2317b26: secret "agc-cred-f09ac2317b26" not found Nov 03 17:06:38.245884 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:38.245887 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv: [secret "agc-cred-b833a8f9cdcf" not found, secret "agc-cred-f09ac2317b26" not found] Nov 03 17:06:38.246168 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:38.245949 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-provider-credentials podName:49219a67-6c62-444a-aaff-4501ec769c84 nodeName:}" failed. No retries permitted until 2026-11-03 17:06:40.245933792 +0000 UTC m=+3143.834933739 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" (UID: "49219a67-6c62-444a-aaff-4501ec769c84") : [secret "agc-cred-b833a8f9cdcf" not found, secret "agc-cred-f09ac2317b26" not found] Nov 03 17:06:38.575597 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:38.575499 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ac31687-2522-4e91-b686-36f48e7f3d15" path="/var/lib/kubelet/pods/4ac31687-2522-4e91-b686-36f48e7f3d15/volumes" Nov 03 17:06:40.277546 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:40.277510 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6046c2/agc-cred-b833a8f9cdcf: secret "agc-cred-b833a8f9cdcf" not found Nov 03 17:06:40.277546 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:40.277544 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6046c2/agc-cred-f09ac2317b26: secret "agc-cred-f09ac2317b26" not found Nov 03 17:06:40.277546 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:40.277553 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv: [secret "agc-cred-b833a8f9cdcf" not found, secret "agc-cred-f09ac2317b26" not found] Nov 03 17:06:40.278092 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:40.277614 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-provider-credentials podName:49219a67-6c62-444a-aaff-4501ec769c84 nodeName:}" failed. No retries permitted until 2026-11-03 17:06:44.277600092 +0000 UTC m=+3147.866600039 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" (UID: "49219a67-6c62-444a-aaff-4501ec769c84") : [secret "agc-cred-b833a8f9cdcf" not found, secret "agc-cred-f09ac2317b26" not found] Nov 03 17:06:43.415835 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:43.415799 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" Nov 03 17:06:43.497190 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:43.497156 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv"] Nov 03 17:06:44.344711 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:44.344656 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6046c2/agc-cred-b833a8f9cdcf: secret "agc-cred-b833a8f9cdcf" not found Nov 03 17:06:44.345013 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:44.344738 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-6046c2/agc-cred-f09ac2317b26: secret "agc-cred-f09ac2317b26" not found Nov 03 17:06:44.345013 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:44.344759 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv: [secret "agc-cred-b833a8f9cdcf" not found, secret "agc-cred-f09ac2317b26" not found] Nov 03 17:06:44.345013 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:44.344857 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-provider-credentials podName:49219a67-6c62-444a-aaff-4501ec769c84 nodeName:}" failed. No retries permitted until 2026-11-03 17:06:52.344837054 +0000 UTC m=+3155.933837004 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" (UID: "49219a67-6c62-444a-aaff-4501ec769c84") : [secret "agc-cred-b833a8f9cdcf" not found, secret "agc-cred-f09ac2317b26" not found] Nov 03 17:06:48.498058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:48.498002 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" podUID="49219a67-6c62-444a-aaff-4501ec769c84" containerName="payload-processing" containerID="cri-o://c6b5a9fc009a76cd550353cb207c95695829f4cf1d0f68c9f473037a52990e98" gracePeriod=25 Nov 03 17:06:48.790859 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:48.790824 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" Nov 03 17:06:48.825219 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:48.825176 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49219a67-6c62-444a-aaff-4501ec769c84-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49219a67-6c62-444a-aaff-4501ec769c84-routing-overlay-volume\") pod \"49219a67-6c62-444a-aaff-4501ec769c84\" (UID: \"49219a67-6c62-444a-aaff-4501ec769c84\") " Nov 03 17:06:48.825398 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:48.825255 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-provider-credentials\" (UniqueName: \"kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-provider-credentials\") pod \"49219a67-6c62-444a-aaff-4501ec769c84\" (UID: \"49219a67-6c62-444a-aaff-4501ec769c84\") " Nov 03 17:06:48.825459 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:48.825442 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49219a67-6c62-444a-aaff-4501ec769c84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49219a67-6c62-444a-aaff-4501ec769c84-plugins-config-volume\") pod \"49219a67-6c62-444a-aaff-4501ec769c84\" (UID: \"49219a67-6c62-444a-aaff-4501ec769c84\") " Nov 03 17:06:48.825587 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:48.825571 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-kube-api-access-7km2h\" (UniqueName: \"kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-kube-api-access-7km2h\") pod \"49219a67-6c62-444a-aaff-4501ec769c84\" (UID: \"49219a67-6c62-444a-aaff-4501ec769c84\") " Nov 03 17:06:48.825649 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:48.825635 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49219a67-6c62-444a-aaff-4501ec769c84-routing-overlay-volume" pod "49219a67-6c62-444a-aaff-4501ec769c84" (UID: "49219a67-6c62-444a-aaff-4501ec769c84"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:06:48.825855 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:48.825832 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49219a67-6c62-444a-aaff-4501ec769c84-plugins-config-volume" pod "49219a67-6c62-444a-aaff-4501ec769c84" (UID: "49219a67-6c62-444a-aaff-4501ec769c84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:06:48.826189 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:48.826171 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49219a67-6c62-444a-aaff-4501ec769c84-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:06:48.826263 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:48.826192 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/49219a67-6c62-444a-aaff-4501ec769c84-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:06:48.827760 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:48.827738 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-provider-credentials" pod "49219a67-6c62-444a-aaff-4501ec769c84" (UID: "49219a67-6c62-444a-aaff-4501ec769c84"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:06:48.827867 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:48.827845 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-kube-api-access-7km2h" pod "49219a67-6c62-444a-aaff-4501ec769c84" (UID: "49219a67-6c62-444a-aaff-4501ec769c84"). InnerVolumeSpecName "kube-api-access-7km2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:06:48.927946 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:48.927902 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7km2h\" (UniqueName: \"kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-kube-api-access-7km2h\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:06:48.927946 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:48.927939 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/49219a67-6c62-444a-aaff-4501ec769c84-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:06:49.561615 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:49.561574 4947 generic.go:356] "Generic (PLEG): container finished" podID="49219a67-6c62-444a-aaff-4501ec769c84" containerID="c6b5a9fc009a76cd550353cb207c95695829f4cf1d0f68c9f473037a52990e98" exitCode=0 Nov 03 17:06:49.562164 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:49.561628 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" Nov 03 17:06:49.562164 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:49.561660 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" event={"ID":"49219a67-6c62-444a-aaff-4501ec769c84","Type":"ContainerDied","Data":"c6b5a9fc009a76cd550353cb207c95695829f4cf1d0f68c9f473037a52990e98"} Nov 03 17:06:49.562164 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:49.561744 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv" event={"ID":"49219a67-6c62-444a-aaff-4501ec769c84","Type":"ContainerDied","Data":"6b617c23f08a67c3496ebe8a931b495a54108d64ccd7cf5de1343771116b9de9"} Nov 03 17:06:49.562164 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:49.561771 4947 scope.go:122] "RemoveContainer" containerID="c6b5a9fc009a76cd550353cb207c95695829f4cf1d0f68c9f473037a52990e98" Nov 03 17:06:49.573471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:49.573448 4947 scope.go:122] "RemoveContainer" containerID="c6b5a9fc009a76cd550353cb207c95695829f4cf1d0f68c9f473037a52990e98" Nov 03 17:06:49.573834 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:06:49.573811 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b5a9fc009a76cd550353cb207c95695829f4cf1d0f68c9f473037a52990e98\": container with ID starting with c6b5a9fc009a76cd550353cb207c95695829f4cf1d0f68c9f473037a52990e98 not found: ID does not exist" containerID="c6b5a9fc009a76cd550353cb207c95695829f4cf1d0f68c9f473037a52990e98" Nov 03 17:06:49.573910 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:49.573846 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b5a9fc009a76cd550353cb207c95695829f4cf1d0f68c9f473037a52990e98"} err="failed to get container status \"c6b5a9fc009a76cd550353cb207c95695829f4cf1d0f68c9f473037a52990e98\": rpc error: code = NotFound desc = could not find container \"c6b5a9fc009a76cd550353cb207c95695829f4cf1d0f68c9f473037a52990e98\": container with ID starting with c6b5a9fc009a76cd550353cb207c95695829f4cf1d0f68c9f473037a52990e98 not found: ID does not exist" Nov 03 17:06:49.623261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:49.623212 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv"] Nov 03 17:06:49.636228 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:49.636193 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-b9547wcwv"] Nov 03 17:06:50.568488 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:06:50.568450 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49219a67-6c62-444a-aaff-4501ec769c84" path="/var/lib/kubelet/pods/49219a67-6c62-444a-aaff-4501ec769c84/volumes" Nov 03 17:07:00.133273 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:00.133236 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:07:10.314447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.314407 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g"] Nov 03 17:07:10.315394 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.315373 4947 cpu_manager.go:414] "containerMap: removing container" podUID="4ac31687-2522-4e91-b686-36f48e7f3d15" containerName="payload-processing" Nov 03 17:07:10.315513 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.315394 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ac31687-2522-4e91-b686-36f48e7f3d15" containerName="payload-processing" Nov 03 17:07:10.315513 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.315412 4947 cpu_manager.go:414] "containerMap: removing container" podUID="49219a67-6c62-444a-aaff-4501ec769c84" containerName="payload-processing" Nov 03 17:07:10.315513 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.315420 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49219a67-6c62-444a-aaff-4501ec769c84" containerName="payload-processing" Nov 03 17:07:10.315671 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.315641 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="49219a67-6c62-444a-aaff-4501ec769c84" containerName="payload-processing" Nov 03 17:07:10.315746 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.315673 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ac31687-2522-4e91-b686-36f48e7f3d15" containerName="payload-processing" Nov 03 17:07:10.321053 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.321029 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" Nov 03 17:07:10.339409 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.339375 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g"] Nov 03 17:07:10.348536 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.348397 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"openshift-service-ca.crt\"" Nov 03 17:07:10.348536 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.348500 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"routing-overlay\"" Nov 03 17:07:10.348832 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.348557 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ca9d7c\"" Nov 03 17:07:10.348832 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.348637 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"kube-root-ca.crt\"" Nov 03 17:07:10.351310 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.351197 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"agc-cred-941b3c888a83\"" Nov 03 17:07:10.351310 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.351278 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"payload-processing-external-model-e2e--d0a79461-dockercfg-c9d8c\"" Nov 03 17:07:10.351458 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.351345 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"agc-cred-76c8387ff3f3\"" Nov 03 17:07:10.351458 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.351398 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"agc-cred-6b6f9f0b9ae4\"" Nov 03 17:07:10.351458 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.351405 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ca9d7c\"/\"agc-cred-4ce9c9bbf166\"" Nov 03 17:07:10.465630 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.465584 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01abc6c1-836a-42bb-8f32-2f1c16d37d27-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g\" (UID: \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" Nov 03 17:07:10.465630 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.465630 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-725vh\" (UniqueName: \"kubernetes.io/projected/01abc6c1-836a-42bb-8f32-2f1c16d37d27-kube-api-access-725vh\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g\" (UID: \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" Nov 03 17:07:10.465968 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.465942 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01abc6c1-836a-42bb-8f32-2f1c16d37d27-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g\" (UID: \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" Nov 03 17:07:10.466027 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.465987 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01abc6c1-836a-42bb-8f32-2f1c16d37d27-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g\" (UID: \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" Nov 03 17:07:10.567291 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.567200 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01abc6c1-836a-42bb-8f32-2f1c16d37d27-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g\" (UID: \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" Nov 03 17:07:10.567291 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.567253 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-725vh\" (UniqueName: \"kubernetes.io/projected/01abc6c1-836a-42bb-8f32-2f1c16d37d27-kube-api-access-725vh\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g\" (UID: \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" Nov 03 17:07:10.567501 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.567431 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01abc6c1-836a-42bb-8f32-2f1c16d37d27-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g\" (UID: \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" Nov 03 17:07:10.567501 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.567457 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01abc6c1-836a-42bb-8f32-2f1c16d37d27-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g\" (UID: \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" Nov 03 17:07:10.567962 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.567932 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01abc6c1-836a-42bb-8f32-2f1c16d37d27-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g\" (UID: \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" Nov 03 17:07:10.568210 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.568191 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01abc6c1-836a-42bb-8f32-2f1c16d37d27-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g\" (UID: \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" Nov 03 17:07:10.570323 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.570297 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01abc6c1-836a-42bb-8f32-2f1c16d37d27-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g\" (UID: \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" Nov 03 17:07:10.611356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.611321 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-725vh\" (UniqueName: \"kubernetes.io/projected/01abc6c1-836a-42bb-8f32-2f1c16d37d27-kube-api-access-725vh\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g\" (UID: \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" Nov 03 17:07:10.638255 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:10.638218 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" Nov 03 17:07:11.052981 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:11.052930 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g"] Nov 03 17:07:11.752374 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:11.752332 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" event={"ID":"01abc6c1-836a-42bb-8f32-2f1c16d37d27","Type":"ContainerStarted","Data":"1412e199693b430a60fd4ded401fb1731454add5e9c24f409f32ba5bdad6908b"} Nov 03 17:07:11.752805 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:11.752389 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" event={"ID":"01abc6c1-836a-42bb-8f32-2f1c16d37d27","Type":"ContainerStarted","Data":"d05f4c8dd42a6ee559b9bf404529ce43889e3c8fb115e739c1251fe715e8887a"} Nov 03 17:07:11.752805 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:11.752410 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" Nov 03 17:07:11.801203 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:11.801139 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" podStartSLOduration=1.801124147 podStartE2EDuration="1.801124147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:07:11.797796921 +0000 UTC m=+3175.386796890" watchObservedRunningTime="2026-11-03 17:07:11.801124147 +0000 UTC m=+3175.390124115" Nov 03 17:07:12.343654 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.343616 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v"] Nov 03 17:07:12.348445 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.348424 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" Nov 03 17:07:12.366926 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.366890 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"agc-cred-3260cfc2e047\"" Nov 03 17:07:12.367118 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.366962 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"agc-cred-1faf98cb4c54\"" Nov 03 17:07:12.367118 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.367087 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e0e63d\"" Nov 03 17:07:12.367241 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.367152 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"agc-cred-0db41f4e4f64\"" Nov 03 17:07:12.380133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.380099 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"kube-root-ca.crt\"" Nov 03 17:07:12.380335 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.380215 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"routing-overlay\"" Nov 03 17:07:12.380589 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.380556 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"agc-cred-bafaaa823370\"" Nov 03 17:07:12.380781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.380612 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"openshift-service-ca.crt\"" Nov 03 17:07:12.380781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.380681 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"payload-processing-external-model-e2e--2172597b-dockercfg-zcksp\"" Nov 03 17:07:12.386106 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.386077 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v"] Nov 03 17:07:12.421986 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.421943 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee041d48-e7d3-4666-97a3-bd9487b18de9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v\" (UID: \"ee041d48-e7d3-4666-97a3-bd9487b18de9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" Nov 03 17:07:12.421986 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.421982 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9xjw\" (UniqueName: \"kubernetes.io/projected/ee041d48-e7d3-4666-97a3-bd9487b18de9-kube-api-access-h9xjw\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v\" (UID: \"ee041d48-e7d3-4666-97a3-bd9487b18de9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" Nov 03 17:07:12.422420 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.422390 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee041d48-e7d3-4666-97a3-bd9487b18de9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v\" (UID: \"ee041d48-e7d3-4666-97a3-bd9487b18de9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" Nov 03 17:07:12.422498 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.422457 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee041d48-e7d3-4666-97a3-bd9487b18de9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v\" (UID: \"ee041d48-e7d3-4666-97a3-bd9487b18de9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" Nov 03 17:07:12.524656 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.524622 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee041d48-e7d3-4666-97a3-bd9487b18de9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v\" (UID: \"ee041d48-e7d3-4666-97a3-bd9487b18de9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" Nov 03 17:07:12.524874 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.524674 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee041d48-e7d3-4666-97a3-bd9487b18de9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v\" (UID: \"ee041d48-e7d3-4666-97a3-bd9487b18de9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" Nov 03 17:07:12.524874 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.524791 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee041d48-e7d3-4666-97a3-bd9487b18de9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v\" (UID: \"ee041d48-e7d3-4666-97a3-bd9487b18de9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" Nov 03 17:07:12.524874 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.524813 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9xjw\" (UniqueName: \"kubernetes.io/projected/ee041d48-e7d3-4666-97a3-bd9487b18de9-kube-api-access-h9xjw\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v\" (UID: \"ee041d48-e7d3-4666-97a3-bd9487b18de9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" Nov 03 17:07:12.525496 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.525468 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee041d48-e7d3-4666-97a3-bd9487b18de9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v\" (UID: \"ee041d48-e7d3-4666-97a3-bd9487b18de9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" Nov 03 17:07:12.525624 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.525501 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee041d48-e7d3-4666-97a3-bd9487b18de9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v\" (UID: \"ee041d48-e7d3-4666-97a3-bd9487b18de9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" Nov 03 17:07:12.528222 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.528197 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee041d48-e7d3-4666-97a3-bd9487b18de9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v\" (UID: \"ee041d48-e7d3-4666-97a3-bd9487b18de9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" Nov 03 17:07:12.550250 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.550213 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9xjw\" (UniqueName: \"kubernetes.io/projected/ee041d48-e7d3-4666-97a3-bd9487b18de9-kube-api-access-h9xjw\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v\" (UID: \"ee041d48-e7d3-4666-97a3-bd9487b18de9\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" Nov 03 17:07:12.664565 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.664461 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" Nov 03 17:07:12.979379 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:07:12.979348 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee041d48_e7d3_4666_97a3_bd9487b18de9.slice/crio-4cd560b143eb241114472cbf580a358a6c1a9d755710fcc2a577012e40e7989f WatchSource:0}: Error finding container 4cd560b143eb241114472cbf580a358a6c1a9d755710fcc2a577012e40e7989f: Status 404 returned error can't find the container with id 4cd560b143eb241114472cbf580a358a6c1a9d755710fcc2a577012e40e7989f Nov 03 17:07:12.981948 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:12.981917 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v"] Nov 03 17:07:13.057903 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:13.057871 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk"] Nov 03 17:07:13.806485 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:13.806438 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" event={"ID":"ee041d48-e7d3-4666-97a3-bd9487b18de9","Type":"ContainerStarted","Data":"f5ae128ad904356940fed5e8bea2a49f2b7296855d81f3450ef1db13ffa34fff"} Nov 03 17:07:13.806748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:13.806598 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" event={"ID":"ee041d48-e7d3-4666-97a3-bd9487b18de9","Type":"ContainerStarted","Data":"4cd560b143eb241114472cbf580a358a6c1a9d755710fcc2a577012e40e7989f"} Nov 03 17:07:13.810750 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:13.807873 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" Nov 03 17:07:13.854683 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:13.854627 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" podStartSLOduration=1.8546120080000001 podStartE2EDuration="1.854612008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:07:13.852483687 +0000 UTC m=+3177.441483656" watchObservedRunningTime="2026-11-03 17:07:13.854612008 +0000 UTC m=+3177.443611995" Nov 03 17:07:18.059066 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.059019 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" podUID="230f71b3-37f5-4ca2-a616-c139cf9b6c93" containerName="payload-processing" containerID="cri-o://a026ac2abc54bf841a9abd9a7c757a44f8aa9276dccea40ac69cda954e202adf" gracePeriod=25 Nov 03 17:07:18.296613 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.296572 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g"] Nov 03 17:07:18.393741 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.393679 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" Nov 03 17:07:18.443534 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.443504 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230f71b3-37f5-4ca2-a616-c139cf9b6c93-kube-api-access-mf4bg\" (UniqueName: \"kubernetes.io/projected/230f71b3-37f5-4ca2-a616-c139cf9b6c93-kube-api-access-mf4bg\") pod \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\" (UID: \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\") " Nov 03 17:07:18.443534 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.443541 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230f71b3-37f5-4ca2-a616-c139cf9b6c93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230f71b3-37f5-4ca2-a616-c139cf9b6c93-plugins-config-volume\") pod \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\" (UID: \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\") " Nov 03 17:07:18.444009 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.443628 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/230f71b3-37f5-4ca2-a616-c139cf9b6c93-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/230f71b3-37f5-4ca2-a616-c139cf9b6c93-routing-overlay-volume\") pod \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\" (UID: \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\") " Nov 03 17:07:18.444009 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.443718 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/230f71b3-37f5-4ca2-a616-c139cf9b6c93-provider-credentials\" (UniqueName: \"kubernetes.io/projected/230f71b3-37f5-4ca2-a616-c139cf9b6c93-provider-credentials\") pod \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\" (UID: \"230f71b3-37f5-4ca2-a616-c139cf9b6c93\") " Nov 03 17:07:18.444194 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.444142 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230f71b3-37f5-4ca2-a616-c139cf9b6c93-plugins-config-volume" pod "230f71b3-37f5-4ca2-a616-c139cf9b6c93" (UID: "230f71b3-37f5-4ca2-a616-c139cf9b6c93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:18.444323 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.444296 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/230f71b3-37f5-4ca2-a616-c139cf9b6c93-routing-overlay-volume" pod "230f71b3-37f5-4ca2-a616-c139cf9b6c93" (UID: "230f71b3-37f5-4ca2-a616-c139cf9b6c93"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:18.444590 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.444567 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/230f71b3-37f5-4ca2-a616-c139cf9b6c93-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:18.444590 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.444593 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/230f71b3-37f5-4ca2-a616-c139cf9b6c93-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:18.446529 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.446500 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230f71b3-37f5-4ca2-a616-c139cf9b6c93-provider-credentials" pod "230f71b3-37f5-4ca2-a616-c139cf9b6c93" (UID: "230f71b3-37f5-4ca2-a616-c139cf9b6c93"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:18.454233 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.454193 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230f71b3-37f5-4ca2-a616-c139cf9b6c93-kube-api-access-mf4bg" pod "230f71b3-37f5-4ca2-a616-c139cf9b6c93" (UID: "230f71b3-37f5-4ca2-a616-c139cf9b6c93"). InnerVolumeSpecName "kube-api-access-mf4bg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:18.546305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.546272 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf4bg\" (UniqueName: \"kubernetes.io/projected/230f71b3-37f5-4ca2-a616-c139cf9b6c93-kube-api-access-mf4bg\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:18.546305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.546302 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/230f71b3-37f5-4ca2-a616-c139cf9b6c93-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:18.902801 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.901103 4947 generic.go:356] "Generic (PLEG): container finished" podID="230f71b3-37f5-4ca2-a616-c139cf9b6c93" containerID="a026ac2abc54bf841a9abd9a7c757a44f8aa9276dccea40ac69cda954e202adf" exitCode=0 Nov 03 17:07:18.902801 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.901354 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" event={"ID":"230f71b3-37f5-4ca2-a616-c139cf9b6c93","Type":"ContainerDied","Data":"a026ac2abc54bf841a9abd9a7c757a44f8aa9276dccea40ac69cda954e202adf"} Nov 03 17:07:18.902801 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.901404 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" event={"ID":"230f71b3-37f5-4ca2-a616-c139cf9b6c93","Type":"ContainerDied","Data":"6856ea6029558e166f51bd3144ec9f00791805a2208c1fae0645d3701e4ecc46"} Nov 03 17:07:18.902801 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.901435 4947 scope.go:122] "RemoveContainer" containerID="a026ac2abc54bf841a9abd9a7c757a44f8aa9276dccea40ac69cda954e202adf" Nov 03 17:07:18.902801 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.901612 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk" Nov 03 17:07:18.921154 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.921071 4947 scope.go:122] "RemoveContainer" containerID="a026ac2abc54bf841a9abd9a7c757a44f8aa9276dccea40ac69cda954e202adf" Nov 03 17:07:18.921487 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:18.921457 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a026ac2abc54bf841a9abd9a7c757a44f8aa9276dccea40ac69cda954e202adf\": container with ID starting with a026ac2abc54bf841a9abd9a7c757a44f8aa9276dccea40ac69cda954e202adf not found: ID does not exist" containerID="a026ac2abc54bf841a9abd9a7c757a44f8aa9276dccea40ac69cda954e202adf" Nov 03 17:07:18.921579 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.921498 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a026ac2abc54bf841a9abd9a7c757a44f8aa9276dccea40ac69cda954e202adf"} err="failed to get container status \"a026ac2abc54bf841a9abd9a7c757a44f8aa9276dccea40ac69cda954e202adf\": rpc error: code = NotFound desc = could not find container \"a026ac2abc54bf841a9abd9a7c757a44f8aa9276dccea40ac69cda954e202adf\": container with ID starting with a026ac2abc54bf841a9abd9a7c757a44f8aa9276dccea40ac69cda954e202adf not found: ID does not exist" Nov 03 17:07:18.967353 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.967306 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk"] Nov 03 17:07:18.972774 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:18.972733 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6b67np7vk"] Nov 03 17:07:20.570120 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:20.570073 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="230f71b3-37f5-4ca2-a616-c139cf9b6c93" path="/var/lib/kubelet/pods/230f71b3-37f5-4ca2-a616-c139cf9b6c93/volumes" Nov 03 17:07:22.582053 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.582021 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j"] Nov 03 17:07:22.582924 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.582900 4947 cpu_manager.go:414] "containerMap: removing container" podUID="230f71b3-37f5-4ca2-a616-c139cf9b6c93" containerName="payload-processing" Nov 03 17:07:22.582924 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.582918 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="230f71b3-37f5-4ca2-a616-c139cf9b6c93" containerName="payload-processing" Nov 03 17:07:22.583086 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.583071 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="230f71b3-37f5-4ca2-a616-c139cf9b6c93" containerName="payload-processing" Nov 03 17:07:22.588275 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.588251 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" Nov 03 17:07:22.606074 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.606032 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a0e27\"/\"agc-cred-6cd2cc145091\"" Nov 03 17:07:22.607913 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.607884 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j"] Nov 03 17:07:22.616612 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.616570 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a0e27\"/\"payload-processing-external-model-e2e--8dd308cd-dockercfg-8qv4x\"" Nov 03 17:07:22.708390 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.708348 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f011e68-26d2-40c9-b439-550813b3fdc9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j\" (UID: \"2f011e68-26d2-40c9-b439-550813b3fdc9\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" Nov 03 17:07:22.708630 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.708515 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rclnj\" (UniqueName: \"kubernetes.io/projected/2f011e68-26d2-40c9-b439-550813b3fdc9-kube-api-access-rclnj\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j\" (UID: \"2f011e68-26d2-40c9-b439-550813b3fdc9\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" Nov 03 17:07:22.708630 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.708555 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f011e68-26d2-40c9-b439-550813b3fdc9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j\" (UID: \"2f011e68-26d2-40c9-b439-550813b3fdc9\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" Nov 03 17:07:22.708630 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.708625 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f011e68-26d2-40c9-b439-550813b3fdc9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j\" (UID: \"2f011e68-26d2-40c9-b439-550813b3fdc9\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" Nov 03 17:07:22.801059 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.801021 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" Nov 03 17:07:22.811730 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.811665 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f011e68-26d2-40c9-b439-550813b3fdc9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j\" (UID: \"2f011e68-26d2-40c9-b439-550813b3fdc9\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" Nov 03 17:07:22.813057 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.813029 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rclnj\" (UniqueName: \"kubernetes.io/projected/2f011e68-26d2-40c9-b439-550813b3fdc9-kube-api-access-rclnj\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j\" (UID: \"2f011e68-26d2-40c9-b439-550813b3fdc9\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" Nov 03 17:07:22.813539 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.813519 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f011e68-26d2-40c9-b439-550813b3fdc9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j\" (UID: \"2f011e68-26d2-40c9-b439-550813b3fdc9\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" Nov 03 17:07:22.837775 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.819780 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f011e68-26d2-40c9-b439-550813b3fdc9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j\" (UID: \"2f011e68-26d2-40c9-b439-550813b3fdc9\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" Nov 03 17:07:22.838344 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.816370 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f011e68-26d2-40c9-b439-550813b3fdc9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j\" (UID: \"2f011e68-26d2-40c9-b439-550813b3fdc9\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" Nov 03 17:07:22.838503 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.812838 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f011e68-26d2-40c9-b439-550813b3fdc9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j\" (UID: \"2f011e68-26d2-40c9-b439-550813b3fdc9\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" Nov 03 17:07:22.950676 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.950564 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f011e68-26d2-40c9-b439-550813b3fdc9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j\" (UID: \"2f011e68-26d2-40c9-b439-550813b3fdc9\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" Nov 03 17:07:22.992277 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:22.990596 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rclnj\" (UniqueName: \"kubernetes.io/projected/2f011e68-26d2-40c9-b439-550813b3fdc9-kube-api-access-rclnj\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j\" (UID: \"2f011e68-26d2-40c9-b439-550813b3fdc9\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" Nov 03 17:07:23.204964 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:23.204878 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" Nov 03 17:07:23.298680 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:23.298593 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" podUID="42eb2139-5c0f-47f3-8c63-dc38a2ff85de" containerName="payload-processing" containerID="cri-o://4f8cf3a1f86c3b1b8c733d6c5ef470cda822437631c7a93c853cab1076b43189" gracePeriod=25 Nov 03 17:07:23.545830 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:23.545797 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j"] Nov 03 17:07:23.633656 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:23.633634 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" Nov 03 17:07:23.758896 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:23.758865 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-plugins-config-volume\") pod \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\" (UID: \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\") " Nov 03 17:07:23.759095 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:23.758935 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-routing-overlay-volume\") pod \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\" (UID: \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\") " Nov 03 17:07:23.759095 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:23.759002 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-provider-credentials\" (UniqueName: \"kubernetes.io/projected/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-provider-credentials\") pod \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\" (UID: \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\") " Nov 03 17:07:23.759196 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:23.759170 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-kube-api-access-vk4w4\" (UniqueName: \"kubernetes.io/projected/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-kube-api-access-vk4w4\") pod \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\" (UID: \"42eb2139-5c0f-47f3-8c63-dc38a2ff85de\") " Nov 03 17:07:23.759354 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:23.759321 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-plugins-config-volume" pod "42eb2139-5c0f-47f3-8c63-dc38a2ff85de" (UID: "42eb2139-5c0f-47f3-8c63-dc38a2ff85de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:23.759444 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:23.759366 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-routing-overlay-volume" pod "42eb2139-5c0f-47f3-8c63-dc38a2ff85de" (UID: "42eb2139-5c0f-47f3-8c63-dc38a2ff85de"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:23.759933 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:23.759847 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:23.759933 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:23.759874 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:23.761461 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:23.761440 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-provider-credentials" pod "42eb2139-5c0f-47f3-8c63-dc38a2ff85de" (UID: "42eb2139-5c0f-47f3-8c63-dc38a2ff85de"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:23.761543 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:23.761493 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-kube-api-access-vk4w4" pod "42eb2139-5c0f-47f3-8c63-dc38a2ff85de" (UID: "42eb2139-5c0f-47f3-8c63-dc38a2ff85de"). InnerVolumeSpecName "kube-api-access-vk4w4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:23.861103 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:23.861016 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:23.861103 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:23.861043 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk4w4\" (UniqueName: \"kubernetes.io/projected/42eb2139-5c0f-47f3-8c63-dc38a2ff85de-kube-api-access-vk4w4\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:24.013417 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:24.013375 4947 generic.go:356] "Generic (PLEG): container finished" podID="42eb2139-5c0f-47f3-8c63-dc38a2ff85de" containerID="4f8cf3a1f86c3b1b8c733d6c5ef470cda822437631c7a93c853cab1076b43189" exitCode=0 Nov 03 17:07:24.013634 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:24.013425 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" Nov 03 17:07:24.013634 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:24.013465 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" event={"ID":"42eb2139-5c0f-47f3-8c63-dc38a2ff85de","Type":"ContainerDied","Data":"4f8cf3a1f86c3b1b8c733d6c5ef470cda822437631c7a93c853cab1076b43189"} Nov 03 17:07:24.013634 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:24.013527 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g" event={"ID":"42eb2139-5c0f-47f3-8c63-dc38a2ff85de","Type":"ContainerDied","Data":"a630c4dfc22ecdc51a06cc48ddec05e7127bd2a9cbc3ef1672cf413d1186f888"} Nov 03 17:07:24.013634 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:24.013556 4947 scope.go:122] "RemoveContainer" containerID="4f8cf3a1f86c3b1b8c733d6c5ef470cda822437631c7a93c853cab1076b43189" Nov 03 17:07:24.015923 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:24.015792 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" event={"ID":"2f011e68-26d2-40c9-b439-550813b3fdc9","Type":"ContainerStarted","Data":"858b7a5f5da2ef0c107e6228973bc37bed51c2c13f0f72b9202d2fda1b1d2350"} Nov 03 17:07:24.015923 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:24.015856 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" event={"ID":"2f011e68-26d2-40c9-b439-550813b3fdc9","Type":"ContainerStarted","Data":"6536b952cee49daa0d87bcc10f67acd52db243f09d0ab5e3fa373fccc18f012b"} Nov 03 17:07:24.015923 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:24.015887 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" Nov 03 17:07:24.025858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:24.025832 4947 scope.go:122] "RemoveContainer" containerID="4f8cf3a1f86c3b1b8c733d6c5ef470cda822437631c7a93c853cab1076b43189" Nov 03 17:07:24.026203 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:24.026181 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f8cf3a1f86c3b1b8c733d6c5ef470cda822437631c7a93c853cab1076b43189\": container with ID starting with 4f8cf3a1f86c3b1b8c733d6c5ef470cda822437631c7a93c853cab1076b43189 not found: ID does not exist" containerID="4f8cf3a1f86c3b1b8c733d6c5ef470cda822437631c7a93c853cab1076b43189" Nov 03 17:07:24.026281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:24.026214 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f8cf3a1f86c3b1b8c733d6c5ef470cda822437631c7a93c853cab1076b43189"} err="failed to get container status \"4f8cf3a1f86c3b1b8c733d6c5ef470cda822437631c7a93c853cab1076b43189\": rpc error: code = NotFound desc = could not find container \"4f8cf3a1f86c3b1b8c733d6c5ef470cda822437631c7a93c853cab1076b43189\": container with ID starting with 4f8cf3a1f86c3b1b8c733d6c5ef470cda822437631c7a93c853cab1076b43189 not found: ID does not exist" Nov 03 17:07:24.082186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:24.082150 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g"] Nov 03 17:07:24.097987 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:24.097949 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-5574r7q5g"] Nov 03 17:07:24.181312 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:24.181204 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" podStartSLOduration=2.181191379 podStartE2EDuration="2.181191379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:07:24.176190564 +0000 UTC m=+3187.765190533" watchObservedRunningTime="2026-11-03 17:07:24.181191379 +0000 UTC m=+3187.770191356" Nov 03 17:07:24.453520 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:24.453443 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj"] Nov 03 17:07:24.453690 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:24.453657 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" podUID="d7425c4a-724e-4835-b34d-47b69913caf2" containerName="istio-proxy" containerID="cri-o://1fab539209cbd79c1513a14ed573f5e8cfffd2e22de651805ddeec112b193d59" gracePeriod=30 Nov 03 17:07:24.583158 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:24.583127 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42eb2139-5c0f-47f3-8c63-dc38a2ff85de" path="/var/lib/kubelet/pods/42eb2139-5c0f-47f3-8c63-dc38a2ff85de/volumes" Nov 03 17:07:25.823778 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:25.823743 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" Nov 03 17:07:27.513323 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:27.513248 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:07:29.776631 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.776602 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:07:29.877348 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.877304 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7425c4a-724e-4835-b34d-47b69913caf2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7425c4a-724e-4835-b34d-47b69913caf2-istiod-ca-cert\") pod \"d7425c4a-724e-4835-b34d-47b69913caf2\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " Nov 03 17:07:29.877959 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.877925 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-istio-envoy\") pod \"d7425c4a-724e-4835-b34d-47b69913caf2\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " Nov 03 17:07:29.878912 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.878873 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7425c4a-724e-4835-b34d-47b69913caf2-kube-api-access-7z86j\" (UniqueName: \"kubernetes.io/projected/d7425c4a-724e-4835-b34d-47b69913caf2-kube-api-access-7z86j\") pod \"d7425c4a-724e-4835-b34d-47b69913caf2\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " Nov 03 17:07:29.879086 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.879018 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7425c4a-724e-4835-b34d-47b69913caf2-istiod-ca-cert" pod "d7425c4a-724e-4835-b34d-47b69913caf2" (UID: "d7425c4a-724e-4835-b34d-47b69913caf2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:29.879329 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.879310 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-workload-certs\") pod \"d7425c4a-724e-4835-b34d-47b69913caf2\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " Nov 03 17:07:29.879442 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.879348 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7425c4a-724e-4835-b34d-47b69913caf2-istio-token\" (UniqueName: \"kubernetes.io/projected/d7425c4a-724e-4835-b34d-47b69913caf2-istio-token\") pod \"d7425c4a-724e-4835-b34d-47b69913caf2\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " Nov 03 17:07:29.879528 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.879492 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7425c4a-724e-4835-b34d-47b69913caf2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7425c4a-724e-4835-b34d-47b69913caf2-istio-podinfo\") pod \"d7425c4a-724e-4835-b34d-47b69913caf2\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " Nov 03 17:07:29.879609 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.879588 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-workload-socket\") pod \"d7425c4a-724e-4835-b34d-47b69913caf2\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " Nov 03 17:07:29.879721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.879647 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-istio-data\") pod \"d7425c4a-724e-4835-b34d-47b69913caf2\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " Nov 03 17:07:29.879721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.879675 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-credential-socket\") pod \"d7425c4a-724e-4835-b34d-47b69913caf2\" (UID: \"d7425c4a-724e-4835-b34d-47b69913caf2\") " Nov 03 17:07:29.880205 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.880155 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-workload-certs" pod "d7425c4a-724e-4835-b34d-47b69913caf2" (UID: "d7425c4a-724e-4835-b34d-47b69913caf2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:07:29.880593 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.880569 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-credential-socket" pod "d7425c4a-724e-4835-b34d-47b69913caf2" (UID: "d7425c4a-724e-4835-b34d-47b69913caf2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:07:29.882431 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.882406 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-workload-socket" pod "d7425c4a-724e-4835-b34d-47b69913caf2" (UID: "d7425c4a-724e-4835-b34d-47b69913caf2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:07:29.883606 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.883567 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-istio-envoy" pod "d7425c4a-724e-4835-b34d-47b69913caf2" (UID: "d7425c4a-724e-4835-b34d-47b69913caf2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:07:29.883816 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.883798 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-istio-data" pod "d7425c4a-724e-4835-b34d-47b69913caf2" (UID: "d7425c4a-724e-4835-b34d-47b69913caf2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:07:29.885176 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.885154 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:29.885360 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.885343 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:29.885507 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.885493 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7425c4a-724e-4835-b34d-47b69913caf2-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:29.885649 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.885621 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:29.885853 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.885839 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:29.885991 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.885962 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7425c4a-724e-4835-b34d-47b69913caf2-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:29.886814 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.886776 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7425c4a-724e-4835-b34d-47b69913caf2-istio-podinfo" pod "d7425c4a-724e-4835-b34d-47b69913caf2" (UID: "d7425c4a-724e-4835-b34d-47b69913caf2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:07:29.890436 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.890402 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7425c4a-724e-4835-b34d-47b69913caf2-kube-api-access-7z86j" pod "d7425c4a-724e-4835-b34d-47b69913caf2" (UID: "d7425c4a-724e-4835-b34d-47b69913caf2"). InnerVolumeSpecName "kube-api-access-7z86j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:29.892025 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.891998 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7425c4a-724e-4835-b34d-47b69913caf2-istio-token" pod "d7425c4a-724e-4835-b34d-47b69913caf2" (UID: "d7425c4a-724e-4835-b34d-47b69913caf2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:29.988069 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.988028 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7425c4a-724e-4835-b34d-47b69913caf2-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:29.988069 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.988072 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z86j\" (UniqueName: \"kubernetes.io/projected/d7425c4a-724e-4835-b34d-47b69913caf2-kube-api-access-7z86j\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:29.988292 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:29.988088 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7425c4a-724e-4835-b34d-47b69913caf2-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:30.089368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:30.089330 4947 generic.go:356] "Generic (PLEG): container finished" podID="d7425c4a-724e-4835-b34d-47b69913caf2" containerID="1fab539209cbd79c1513a14ed573f5e8cfffd2e22de651805ddeec112b193d59" exitCode=0 Nov 03 17:07:30.089565 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:30.089403 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" Nov 03 17:07:30.089565 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:30.089416 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" event={"ID":"d7425c4a-724e-4835-b34d-47b69913caf2","Type":"ContainerDied","Data":"1fab539209cbd79c1513a14ed573f5e8cfffd2e22de651805ddeec112b193d59"} Nov 03 17:07:30.089565 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:30.089498 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj" event={"ID":"d7425c4a-724e-4835-b34d-47b69913caf2","Type":"ContainerDied","Data":"d97b91c417943c5350891dade707c544f3b7c7298d6bb3e3f21f175a49817273"} Nov 03 17:07:30.089565 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:30.089523 4947 scope.go:122] "RemoveContainer" containerID="1fab539209cbd79c1513a14ed573f5e8cfffd2e22de651805ddeec112b193d59" Nov 03 17:07:30.101427 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:30.101398 4947 scope.go:122] "RemoveContainer" containerID="1fab539209cbd79c1513a14ed573f5e8cfffd2e22de651805ddeec112b193d59" Nov 03 17:07:30.101845 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:30.101819 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fab539209cbd79c1513a14ed573f5e8cfffd2e22de651805ddeec112b193d59\": container with ID starting with 1fab539209cbd79c1513a14ed573f5e8cfffd2e22de651805ddeec112b193d59 not found: ID does not exist" containerID="1fab539209cbd79c1513a14ed573f5e8cfffd2e22de651805ddeec112b193d59" Nov 03 17:07:30.101962 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:30.101859 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fab539209cbd79c1513a14ed573f5e8cfffd2e22de651805ddeec112b193d59"} err="failed to get container status \"1fab539209cbd79c1513a14ed573f5e8cfffd2e22de651805ddeec112b193d59\": rpc error: code = NotFound desc = could not find container \"1fab539209cbd79c1513a14ed573f5e8cfffd2e22de651805ddeec112b193d59\": container with ID starting with 1fab539209cbd79c1513a14ed573f5e8cfffd2e22de651805ddeec112b193d59 not found: ID does not exist" Nov 03 17:07:30.133209 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:30.133120 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:07:30.188620 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:30.188586 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj"] Nov 03 17:07:30.193764 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:30.193731 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-bxtlj"] Nov 03 17:07:30.621288 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:30.621254 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7425c4a-724e-4835-b34d-47b69913caf2" path="/var/lib/kubelet/pods/d7425c4a-724e-4835-b34d-47b69913caf2/volumes" Nov 03 17:07:32.180086 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.178011 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8"] Nov 03 17:07:32.191729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.184023 4947 cpu_manager.go:414] "containerMap: removing container" podUID="d7425c4a-724e-4835-b34d-47b69913caf2" containerName="istio-proxy" Nov 03 17:07:32.191729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.184050 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7425c4a-724e-4835-b34d-47b69913caf2" containerName="istio-proxy" Nov 03 17:07:32.191729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.184096 4947 cpu_manager.go:414] "containerMap: removing container" podUID="42eb2139-5c0f-47f3-8c63-dc38a2ff85de" containerName="payload-processing" Nov 03 17:07:32.191729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.184102 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42eb2139-5c0f-47f3-8c63-dc38a2ff85de" containerName="payload-processing" Nov 03 17:07:32.191729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.184276 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7425c4a-724e-4835-b34d-47b69913caf2" containerName="istio-proxy" Nov 03 17:07:32.191729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.184294 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="42eb2139-5c0f-47f3-8c63-dc38a2ff85de" containerName="payload-processing" Nov 03 17:07:32.198733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.194129 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" Nov 03 17:07:32.198733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.198496 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8"] Nov 03 17:07:32.220779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.217176 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"agc-cred-4fb82f7c0e56\"" Nov 03 17:07:32.237758 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.230366 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"kube-root-ca.crt\"" Nov 03 17:07:32.237758 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.230475 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"payload-processing-external-model-e2e--6d96ad03-dockercfg-wrw88\"" Nov 03 17:07:32.237758 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.230595 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2492c4\"" Nov 03 17:07:32.237758 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.230657 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"openshift-service-ca.crt\"" Nov 03 17:07:32.238268 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.238238 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"routing-overlay\"" Nov 03 17:07:32.289723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.283068 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8\" (UID: \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" Nov 03 17:07:32.289723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.283301 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npg8f\" (UniqueName: \"kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-kube-api-access-npg8f\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8\" (UID: \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" Nov 03 17:07:32.289723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.283465 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8\" (UID: \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" Nov 03 17:07:32.289723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.283557 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8\" (UID: \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" Nov 03 17:07:32.323733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.323522 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s"] Nov 03 17:07:32.390385 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.390107 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npg8f\" (UniqueName: \"kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-kube-api-access-npg8f\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8\" (UID: \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" Nov 03 17:07:32.390385 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.390301 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8\" (UID: \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" Nov 03 17:07:32.390385 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.390344 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8\" (UID: \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" Nov 03 17:07:32.390738 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.390462 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8\" (UID: \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" Nov 03 17:07:32.393205 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.392970 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8\" (UID: \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" Nov 03 17:07:32.393205 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.393122 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8\" (UID: \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" Nov 03 17:07:32.394446 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.394419 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8\" (UID: \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" Nov 03 17:07:32.431674 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.431597 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npg8f\" (UniqueName: \"kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-kube-api-access-npg8f\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8\" (UID: \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" Nov 03 17:07:32.566779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.566740 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" Nov 03 17:07:32.885028 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:07:32.884997 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafd6cbf7_0712_49f1_96ae_f2fcb0bf94bd.slice/crio-a0b1366dda826cba52c90fbc056e5fc594d12d80bfaa3728cf700b1adcfee022 WatchSource:0}: Error finding container a0b1366dda826cba52c90fbc056e5fc594d12d80bfaa3728cf700b1adcfee022: Status 404 returned error can't find the container with id a0b1366dda826cba52c90fbc056e5fc594d12d80bfaa3728cf700b1adcfee022 Nov 03 17:07:32.885740 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:32.885501 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8"] Nov 03 17:07:33.261723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:33.259646 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" event={"ID":"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd","Type":"ContainerStarted","Data":"ca9a1977d9fccc11b720ab4a511de25d780203395e9b58385b566ed59e6ffc2b"} Nov 03 17:07:33.261723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:33.259742 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" event={"ID":"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd","Type":"ContainerStarted","Data":"a0b1366dda826cba52c90fbc056e5fc594d12d80bfaa3728cf700b1adcfee022"} Nov 03 17:07:33.261723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:33.261673 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" Nov 03 17:07:33.412721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:33.411937 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" podStartSLOduration=1.411911265 podStartE2EDuration="1.411911265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:07:33.308176059 +0000 UTC m=+3196.897176040" watchObservedRunningTime="2026-11-03 17:07:33.411911265 +0000 UTC m=+3197.000911235" Nov 03 17:07:33.444724 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:33.440890 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2492c4/agc-cred-4fb82f7c0e56: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:07:33.444724 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:33.440923 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:07:33.456719 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:33.450799 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials podName:afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd nodeName:}" failed. No retries permitted until 2026-11-03 17:07:33.950757821 +0000 UTC m=+3197.539757777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" (UID: "afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd") : secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:07:33.961473 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:33.961443 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2492c4/agc-cred-4fb82f7c0e56: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:07:33.961473 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:33.961467 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:07:33.961787 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:33.961527 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials podName:afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd nodeName:}" failed. No retries permitted until 2026-11-03 17:07:34.961509434 +0000 UTC m=+3198.550509383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" (UID: "afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd") : secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:07:34.110250 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:34.110208 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl"] Nov 03 17:07:34.116750 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:34.116692 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" Nov 03 17:07:34.132843 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:34.132813 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"e2e-cfl-c95486-provider-cred\"" Nov 03 17:07:34.134518 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:34.134488 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl"] Nov 03 17:07:34.170147 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:34.170091 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpflr\" (UniqueName: \"kubernetes.io/projected/d4cda53a-2726-4dc0-8057-f3302901c980-kube-api-access-lpflr\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl\" (UID: \"d4cda53a-2726-4dc0-8057-f3302901c980\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" Nov 03 17:07:34.170363 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:34.170201 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4cda53a-2726-4dc0-8057-f3302901c980-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl\" (UID: \"d4cda53a-2726-4dc0-8057-f3302901c980\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" Nov 03 17:07:34.170507 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:34.170486 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cda53a-2726-4dc0-8057-f3302901c980-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl\" (UID: \"d4cda53a-2726-4dc0-8057-f3302901c980\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" Nov 03 17:07:34.170569 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:34.170526 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4cda53a-2726-4dc0-8057-f3302901c980-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl\" (UID: \"d4cda53a-2726-4dc0-8057-f3302901c980\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" Nov 03 17:07:34.272009 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:34.271972 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4cda53a-2726-4dc0-8057-f3302901c980-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl\" (UID: \"d4cda53a-2726-4dc0-8057-f3302901c980\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" Nov 03 17:07:34.272448 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:34.272191 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cda53a-2726-4dc0-8057-f3302901c980-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl\" (UID: \"d4cda53a-2726-4dc0-8057-f3302901c980\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" Nov 03 17:07:34.272448 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:34.272221 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4cda53a-2726-4dc0-8057-f3302901c980-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl\" (UID: \"d4cda53a-2726-4dc0-8057-f3302901c980\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" Nov 03 17:07:34.272448 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:34.272366 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpflr\" (UniqueName: \"kubernetes.io/projected/d4cda53a-2726-4dc0-8057-f3302901c980-kube-api-access-lpflr\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl\" (UID: \"d4cda53a-2726-4dc0-8057-f3302901c980\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" Nov 03 17:07:34.272997 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:34.272969 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4cda53a-2726-4dc0-8057-f3302901c980-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl\" (UID: \"d4cda53a-2726-4dc0-8057-f3302901c980\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" Nov 03 17:07:34.273092 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:34.272995 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cda53a-2726-4dc0-8057-f3302901c980-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl\" (UID: \"d4cda53a-2726-4dc0-8057-f3302901c980\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" Nov 03 17:07:34.274785 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:34.274756 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4cda53a-2726-4dc0-8057-f3302901c980-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl\" (UID: \"d4cda53a-2726-4dc0-8057-f3302901c980\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" Nov 03 17:07:34.318138 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:34.318088 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpflr\" (UniqueName: \"kubernetes.io/projected/d4cda53a-2726-4dc0-8057-f3302901c980-kube-api-access-lpflr\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl\" (UID: \"d4cda53a-2726-4dc0-8057-f3302901c980\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" Nov 03 17:07:34.440723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:34.440150 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" Nov 03 17:07:34.972862 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:34.972819 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl"] Nov 03 17:07:34.973997 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:07:34.973964 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4cda53a_2726_4dc0_8057_f3302901c980.slice/crio-c58c44f2b768147e52443f2ef2e9daabc01009348c15aec18f01efd357735117 WatchSource:0}: Error finding container c58c44f2b768147e52443f2ef2e9daabc01009348c15aec18f01efd357735117: Status 404 returned error can't find the container with id c58c44f2b768147e52443f2ef2e9daabc01009348c15aec18f01efd357735117 Nov 03 17:07:35.007772 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:35.007738 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2492c4/agc-cred-4fb82f7c0e56: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:07:35.007772 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:35.007765 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:07:35.007951 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:35.007830 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials podName:afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd nodeName:}" failed. No retries permitted until 2026-11-03 17:07:37.007813739 +0000 UTC m=+3200.596813687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" (UID: "afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd") : secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:07:35.041878 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.041827 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" Nov 03 17:07:35.379004 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.378960 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" event={"ID":"d4cda53a-2726-4dc0-8057-f3302901c980","Type":"ContainerStarted","Data":"3db109b67775216034362248e9f2cbbbe3e3db20e4a78582099f7cafe1f1a413"} Nov 03 17:07:35.379502 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.379037 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" event={"ID":"d4cda53a-2726-4dc0-8057-f3302901c980","Type":"ContainerStarted","Data":"c58c44f2b768147e52443f2ef2e9daabc01009348c15aec18f01efd357735117"} Nov 03 17:07:35.379502 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.379069 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" Nov 03 17:07:35.429141 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.429101 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg"] Nov 03 17:07:35.433396 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.433338 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" podStartSLOduration=1.433322384 podStartE2EDuration="1.433322384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:07:35.426937568 +0000 UTC m=+3199.015937539" watchObservedRunningTime="2026-11-03 17:07:35.433322384 +0000 UTC m=+3199.022322355" Nov 03 17:07:35.435232 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.435211 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" Nov 03 17:07:35.452104 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.452071 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1f83c0\"/\"payload-processing-external-model-e2e--91a4f230-dockercfg-dm6c8\"" Nov 03 17:07:35.452803 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.452779 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1f83c0\"/\"agc-cred-0874b9379bdd\"" Nov 03 17:07:35.459246 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.458103 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg"] Nov 03 17:07:35.625415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.625369 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e624cd3f-1170-4a5f-8f14-9aa63055dce6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg\" (UID: \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" Nov 03 17:07:35.625815 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.625791 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e624cd3f-1170-4a5f-8f14-9aa63055dce6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg\" (UID: \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" Nov 03 17:07:35.626001 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.625982 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwlg2\" (UniqueName: \"kubernetes.io/projected/e624cd3f-1170-4a5f-8f14-9aa63055dce6-kube-api-access-kwlg2\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg\" (UID: \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" Nov 03 17:07:35.626162 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.626144 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e624cd3f-1170-4a5f-8f14-9aa63055dce6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg\" (UID: \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" Nov 03 17:07:35.733815 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.728528 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e624cd3f-1170-4a5f-8f14-9aa63055dce6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg\" (UID: \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" Nov 03 17:07:35.733815 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.728632 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwlg2\" (UniqueName: \"kubernetes.io/projected/e624cd3f-1170-4a5f-8f14-9aa63055dce6-kube-api-access-kwlg2\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg\" (UID: \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" Nov 03 17:07:35.733815 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.728682 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e624cd3f-1170-4a5f-8f14-9aa63055dce6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg\" (UID: \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" Nov 03 17:07:35.733815 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.728964 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e624cd3f-1170-4a5f-8f14-9aa63055dce6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg\" (UID: \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" Nov 03 17:07:35.733815 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.730241 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e624cd3f-1170-4a5f-8f14-9aa63055dce6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg\" (UID: \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" Nov 03 17:07:35.733815 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.730896 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e624cd3f-1170-4a5f-8f14-9aa63055dce6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg\" (UID: \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" Nov 03 17:07:35.740362 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.740092 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e624cd3f-1170-4a5f-8f14-9aa63055dce6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg\" (UID: \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" Nov 03 17:07:35.767582 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.767514 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwlg2\" (UniqueName: \"kubernetes.io/projected/e624cd3f-1170-4a5f-8f14-9aa63055dce6-kube-api-access-kwlg2\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg\" (UID: \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" Nov 03 17:07:35.773725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:35.770010 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" Nov 03 17:07:36.127942 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:36.127913 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg"] Nov 03 17:07:36.387244 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:36.387142 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" event={"ID":"e624cd3f-1170-4a5f-8f14-9aa63055dce6","Type":"ContainerStarted","Data":"6cb3f8aeb445b1acc6b926ef8895f73218b383f84e233dfdf99443297e235e91"} Nov 03 17:07:36.387682 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:36.387427 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" Nov 03 17:07:36.387682 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:36.387472 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" event={"ID":"e624cd3f-1170-4a5f-8f14-9aa63055dce6","Type":"ContainerStarted","Data":"cdfbed753e072f64e4bfec1803214e28917d840cd78e1c62b76f5d78b12a287e"} Nov 03 17:07:36.436531 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:36.436469 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" podStartSLOduration=1.436450384 podStartE2EDuration="1.436450384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:07:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:07:36.432383309 +0000 UTC m=+3200.021383279" watchObservedRunningTime="2026-11-03 17:07:36.436450384 +0000 UTC m=+3200.025450374" Nov 03 17:07:37.050403 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:37.050359 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2492c4/agc-cred-4fb82f7c0e56: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:07:37.050403 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:37.050398 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:07:37.050684 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:37.050477 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials podName:afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd nodeName:}" failed. No retries permitted until 2026-11-03 17:07:41.050455958 +0000 UTC m=+3204.639455920 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" (UID: "afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd") : secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:07:37.324844 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:37.324717 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" podUID="c7818056-d43b-4560-9e3f-ad309fd0dd9d" containerName="payload-processing" containerID="cri-o://750da36bdc2c4f089b7580f6ac2f29f462317b55e3d7a575a46d95485b49da04" gracePeriod=25 Nov 03 17:07:37.615789 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:37.615756 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" Nov 03 17:07:37.658160 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:37.658124 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7818056-d43b-4560-9e3f-ad309fd0dd9d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c7818056-d43b-4560-9e3f-ad309fd0dd9d-routing-overlay-volume\") pod \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\" (UID: \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\") " Nov 03 17:07:37.658339 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:37.658264 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7818056-d43b-4560-9e3f-ad309fd0dd9d-kube-api-access-7tlh2\" (UniqueName: \"kubernetes.io/projected/c7818056-d43b-4560-9e3f-ad309fd0dd9d-kube-api-access-7tlh2\") pod \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\" (UID: \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\") " Nov 03 17:07:37.658339 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:37.658284 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7818056-d43b-4560-9e3f-ad309fd0dd9d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c7818056-d43b-4560-9e3f-ad309fd0dd9d-provider-credentials\") pod \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\" (UID: \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\") " Nov 03 17:07:37.658444 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:37.658429 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7818056-d43b-4560-9e3f-ad309fd0dd9d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7818056-d43b-4560-9e3f-ad309fd0dd9d-plugins-config-volume\") pod \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\" (UID: \"c7818056-d43b-4560-9e3f-ad309fd0dd9d\") " Nov 03 17:07:37.658574 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:37.658524 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7818056-d43b-4560-9e3f-ad309fd0dd9d-routing-overlay-volume" pod "c7818056-d43b-4560-9e3f-ad309fd0dd9d" (UID: "c7818056-d43b-4560-9e3f-ad309fd0dd9d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:37.658928 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:37.658883 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7818056-d43b-4560-9e3f-ad309fd0dd9d-plugins-config-volume" pod "c7818056-d43b-4560-9e3f-ad309fd0dd9d" (UID: "c7818056-d43b-4560-9e3f-ad309fd0dd9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:37.659182 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:37.659164 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c7818056-d43b-4560-9e3f-ad309fd0dd9d-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:37.659252 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:37.659185 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7818056-d43b-4560-9e3f-ad309fd0dd9d-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:37.661021 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:37.660996 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7818056-d43b-4560-9e3f-ad309fd0dd9d-kube-api-access-7tlh2" pod "c7818056-d43b-4560-9e3f-ad309fd0dd9d" (UID: "c7818056-d43b-4560-9e3f-ad309fd0dd9d"). InnerVolumeSpecName "kube-api-access-7tlh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:37.661728 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:37.661431 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7818056-d43b-4560-9e3f-ad309fd0dd9d-provider-credentials" pod "c7818056-d43b-4560-9e3f-ad309fd0dd9d" (UID: "c7818056-d43b-4560-9e3f-ad309fd0dd9d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:37.761306 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:37.761270 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tlh2\" (UniqueName: \"kubernetes.io/projected/c7818056-d43b-4560-9e3f-ad309fd0dd9d-kube-api-access-7tlh2\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:37.761306 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:37.761301 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c7818056-d43b-4560-9e3f-ad309fd0dd9d-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:38.404656 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:38.404618 4947 generic.go:356] "Generic (PLEG): container finished" podID="c7818056-d43b-4560-9e3f-ad309fd0dd9d" containerID="750da36bdc2c4f089b7580f6ac2f29f462317b55e3d7a575a46d95485b49da04" exitCode=0 Nov 03 17:07:38.404858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:38.404666 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" Nov 03 17:07:38.404858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:38.404711 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" event={"ID":"c7818056-d43b-4560-9e3f-ad309fd0dd9d","Type":"ContainerDied","Data":"750da36bdc2c4f089b7580f6ac2f29f462317b55e3d7a575a46d95485b49da04"} Nov 03 17:07:38.404858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:38.404764 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s" event={"ID":"c7818056-d43b-4560-9e3f-ad309fd0dd9d","Type":"ContainerDied","Data":"f516959586ea029dbec641f2c71cc93b2dffd783603b608bd6233d1afe209a6c"} Nov 03 17:07:38.404858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:38.404799 4947 scope.go:122] "RemoveContainer" containerID="750da36bdc2c4f089b7580f6ac2f29f462317b55e3d7a575a46d95485b49da04" Nov 03 17:07:38.416902 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:38.416831 4947 scope.go:122] "RemoveContainer" containerID="750da36bdc2c4f089b7580f6ac2f29f462317b55e3d7a575a46d95485b49da04" Nov 03 17:07:38.417145 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:38.417126 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750da36bdc2c4f089b7580f6ac2f29f462317b55e3d7a575a46d95485b49da04\": container with ID starting with 750da36bdc2c4f089b7580f6ac2f29f462317b55e3d7a575a46d95485b49da04 not found: ID does not exist" containerID="750da36bdc2c4f089b7580f6ac2f29f462317b55e3d7a575a46d95485b49da04" Nov 03 17:07:38.417193 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:38.417155 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750da36bdc2c4f089b7580f6ac2f29f462317b55e3d7a575a46d95485b49da04"} err="failed to get container status \"750da36bdc2c4f089b7580f6ac2f29f462317b55e3d7a575a46d95485b49da04\": rpc error: code = NotFound desc = could not find container \"750da36bdc2c4f089b7580f6ac2f29f462317b55e3d7a575a46d95485b49da04\": container with ID starting with 750da36bdc2c4f089b7580f6ac2f29f462317b55e3d7a575a46d95485b49da04 not found: ID does not exist" Nov 03 17:07:38.462994 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:38.462962 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s"] Nov 03 17:07:38.476092 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:38.476060 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-87d6r6n4s"] Nov 03 17:07:38.570567 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:38.570536 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7818056-d43b-4560-9e3f-ad309fd0dd9d" path="/var/lib/kubelet/pods/c7818056-d43b-4560-9e3f-ad309fd0dd9d/volumes" Nov 03 17:07:40.917426 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:40.917388 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8"] Nov 03 17:07:40.918596 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:40.918562 4947 cpu_manager.go:414] "containerMap: removing container" podUID="c7818056-d43b-4560-9e3f-ad309fd0dd9d" containerName="payload-processing" Nov 03 17:07:40.918596 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:40.918591 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7818056-d43b-4560-9e3f-ad309fd0dd9d" containerName="payload-processing" Nov 03 17:07:40.918795 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:40.918781 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7818056-d43b-4560-9e3f-ad309fd0dd9d" containerName="payload-processing" Nov 03 17:07:40.924596 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:40.924570 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" Nov 03 17:07:40.944524 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:40.944490 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8"] Nov 03 17:07:40.952184 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:40.951811 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1f83c0\"/\"agc-cred-cb769e926784\"" Nov 03 17:07:41.027428 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:41.027390 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzmdh\" (UniqueName: \"kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-kube-api-access-nzmdh\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8\" (UID: \"80ffba80-112d-45c3-b127-40a468224b7b\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" Nov 03 17:07:41.027601 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:41.027490 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ffba80-112d-45c3-b127-40a468224b7b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8\" (UID: \"80ffba80-112d-45c3-b127-40a468224b7b\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" Nov 03 17:07:41.027686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:41.027598 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8\" (UID: \"80ffba80-112d-45c3-b127-40a468224b7b\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" Nov 03 17:07:41.027686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:41.027664 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80ffba80-112d-45c3-b127-40a468224b7b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8\" (UID: \"80ffba80-112d-45c3-b127-40a468224b7b\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" Nov 03 17:07:41.129644 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:41.129604 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8\" (UID: \"80ffba80-112d-45c3-b127-40a468224b7b\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" Nov 03 17:07:41.129893 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:41.129687 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80ffba80-112d-45c3-b127-40a468224b7b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8\" (UID: \"80ffba80-112d-45c3-b127-40a468224b7b\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" Nov 03 17:07:41.130046 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:41.130012 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzmdh\" (UniqueName: \"kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-kube-api-access-nzmdh\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8\" (UID: \"80ffba80-112d-45c3-b127-40a468224b7b\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" Nov 03 17:07:41.130217 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:41.130142 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ffba80-112d-45c3-b127-40a468224b7b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8\" (UID: \"80ffba80-112d-45c3-b127-40a468224b7b\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" Nov 03 17:07:41.130217 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:41.130173 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2492c4/agc-cred-4fb82f7c0e56: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:07:41.130217 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:41.130196 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:07:41.130381 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:41.130255 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials podName:afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd nodeName:}" failed. No retries permitted until 2026-11-03 17:07:49.130238812 +0000 UTC m=+3212.719238781 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" (UID: "afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd") : secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:07:41.130625 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:41.130599 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80ffba80-112d-45c3-b127-40a468224b7b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8\" (UID: \"80ffba80-112d-45c3-b127-40a468224b7b\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" Nov 03 17:07:41.131462 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:41.131431 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ffba80-112d-45c3-b127-40a468224b7b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8\" (UID: \"80ffba80-112d-45c3-b127-40a468224b7b\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" Nov 03 17:07:41.132851 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:41.132833 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8\" (UID: \"80ffba80-112d-45c3-b127-40a468224b7b\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" Nov 03 17:07:41.154845 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:41.154817 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzmdh\" (UniqueName: \"kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-kube-api-access-nzmdh\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8\" (UID: \"80ffba80-112d-45c3-b127-40a468224b7b\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" Nov 03 17:07:41.241193 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:41.241163 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" Nov 03 17:07:41.560226 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:07:41.560173 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ffba80_112d_45c3_b127_40a468224b7b.slice/crio-6b17e6b4d22e7883d5f571ff12bc5e8ae5a589505ebe39ae5ea59c25374bbf78 WatchSource:0}: Error finding container 6b17e6b4d22e7883d5f571ff12bc5e8ae5a589505ebe39ae5ea59c25374bbf78: Status 404 returned error can't find the container with id 6b17e6b4d22e7883d5f571ff12bc5e8ae5a589505ebe39ae5ea59c25374bbf78 Nov 03 17:07:41.562157 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:41.562127 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8"] Nov 03 17:07:42.437955 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:42.437920 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" event={"ID":"80ffba80-112d-45c3-b127-40a468224b7b","Type":"ContainerStarted","Data":"a06d3f4be3dc16f1a28c3805a4f91d0573d41fcc1917e6d8095641bcbb798bbc"} Nov 03 17:07:42.438403 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:42.438002 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" event={"ID":"80ffba80-112d-45c3-b127-40a468224b7b","Type":"ContainerStarted","Data":"6b17e6b4d22e7883d5f571ff12bc5e8ae5a589505ebe39ae5ea59c25374bbf78"} Nov 03 17:07:42.438403 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:42.438188 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" Nov 03 17:07:42.486675 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:42.486616 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" podStartSLOduration=2.486602072 podStartE2EDuration="2.486602072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:07:42.483758788 +0000 UTC m=+3206.072758756" watchObservedRunningTime="2026-11-03 17:07:42.486602072 +0000 UTC m=+3206.075602041" Nov 03 17:07:45.382771 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:45.382742 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" Nov 03 17:07:46.390743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:46.390690 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" Nov 03 17:07:46.481259 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:46.481224 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8"] Nov 03 17:07:47.399404 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:47.399376 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" Nov 03 17:07:49.191034 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:49.191003 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2492c4/agc-cred-4fb82f7c0e56: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:07:49.191034 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:49.191030 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:07:49.191600 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:49.191102 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials podName:afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd nodeName:}" failed. No retries permitted until 2026-11-03 17:08:05.191080889 +0000 UTC m=+3228.780080835 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" (UID: "afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd") : secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:07:51.482193 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:51.482127 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" podUID="afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd" containerName="payload-processing" containerID="cri-o://ca9a1977d9fccc11b720ab4a511de25d780203395e9b58385b566ed59e6ffc2b" gracePeriod=25 Nov 03 17:07:51.776973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:51.776948 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" Nov 03 17:07:51.843104 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:51.843069 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials\") pod \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\" (UID: \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\") " Nov 03 17:07:51.843295 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:51.843168 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-routing-overlay-volume\") pod \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\" (UID: \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\") " Nov 03 17:07:51.843364 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:51.843322 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-plugins-config-volume\") pod \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\" (UID: \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\") " Nov 03 17:07:51.843441 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:51.843422 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-kube-api-access-npg8f\" (UniqueName: \"kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-kube-api-access-npg8f\") pod \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\" (UID: \"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd\") " Nov 03 17:07:51.843887 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:51.843856 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-routing-overlay-volume" pod "afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd" (UID: "afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:51.844385 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:51.844009 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-plugins-config-volume" pod "afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd" (UID: "afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:51.844385 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:51.844224 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:51.844385 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:51.844246 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:51.846466 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:51.846439 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-kube-api-access-npg8f" pod "afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd" (UID: "afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd"). InnerVolumeSpecName "kube-api-access-npg8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:51.846880 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:51.846850 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials" pod "afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd" (UID: "afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:51.951447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:51.951405 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npg8f\" (UniqueName: \"kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-kube-api-access-npg8f\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:51.951447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:51.951449 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:52.564790 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:52.564742 4947 generic.go:356] "Generic (PLEG): container finished" podID="afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd" containerID="ca9a1977d9fccc11b720ab4a511de25d780203395e9b58385b566ed59e6ffc2b" exitCode=0 Nov 03 17:07:52.565226 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:52.564873 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" Nov 03 17:07:52.567961 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:52.567938 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" event={"ID":"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd","Type":"ContainerDied","Data":"ca9a1977d9fccc11b720ab4a511de25d780203395e9b58385b566ed59e6ffc2b"} Nov 03 17:07:52.568059 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:52.567983 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8" event={"ID":"afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd","Type":"ContainerDied","Data":"a0b1366dda826cba52c90fbc056e5fc594d12d80bfaa3728cf700b1adcfee022"} Nov 03 17:07:52.568059 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:52.568015 4947 scope.go:122] "RemoveContainer" containerID="ca9a1977d9fccc11b720ab4a511de25d780203395e9b58385b566ed59e6ffc2b" Nov 03 17:07:52.580576 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:52.580555 4947 scope.go:122] "RemoveContainer" containerID="ca9a1977d9fccc11b720ab4a511de25d780203395e9b58385b566ed59e6ffc2b" Nov 03 17:07:52.580946 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:52.580927 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9a1977d9fccc11b720ab4a511de25d780203395e9b58385b566ed59e6ffc2b\": container with ID starting with ca9a1977d9fccc11b720ab4a511de25d780203395e9b58385b566ed59e6ffc2b not found: ID does not exist" containerID="ca9a1977d9fccc11b720ab4a511de25d780203395e9b58385b566ed59e6ffc2b" Nov 03 17:07:52.581001 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:52.580953 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9a1977d9fccc11b720ab4a511de25d780203395e9b58385b566ed59e6ffc2b"} err="failed to get container status \"ca9a1977d9fccc11b720ab4a511de25d780203395e9b58385b566ed59e6ffc2b\": rpc error: code = NotFound desc = could not find container \"ca9a1977d9fccc11b720ab4a511de25d780203395e9b58385b566ed59e6ffc2b\": container with ID starting with ca9a1977d9fccc11b720ab4a511de25d780203395e9b58385b566ed59e6ffc2b not found: ID does not exist" Nov 03 17:07:52.623252 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:52.623219 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8"] Nov 03 17:07:52.636736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:52.636686 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5dbst8"] Nov 03 17:07:53.449757 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:53.449729 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" Nov 03 17:07:53.531589 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:53.531555 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg"] Nov 03 17:07:54.567491 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:54.567458 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd" path="/var/lib/kubelet/pods/afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd/volumes" Nov 03 17:07:58.532054 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:58.532013 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" podUID="e624cd3f-1170-4a5f-8f14-9aa63055dce6" containerName="payload-processing" containerID="cri-o://6cb3f8aeb445b1acc6b926ef8895f73218b383f84e233dfdf99443297e235e91" gracePeriod=25 Nov 03 17:07:58.828410 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:58.828383 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" Nov 03 17:07:58.959477 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:58.959445 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e624cd3f-1170-4a5f-8f14-9aa63055dce6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e624cd3f-1170-4a5f-8f14-9aa63055dce6-plugins-config-volume\") pod \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\" (UID: \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\") " Nov 03 17:07:58.959690 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:58.959497 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e624cd3f-1170-4a5f-8f14-9aa63055dce6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e624cd3f-1170-4a5f-8f14-9aa63055dce6-routing-overlay-volume\") pod \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\" (UID: \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\") " Nov 03 17:07:58.959690 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:58.959626 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e624cd3f-1170-4a5f-8f14-9aa63055dce6-kube-api-access-kwlg2\" (UniqueName: \"kubernetes.io/projected/e624cd3f-1170-4a5f-8f14-9aa63055dce6-kube-api-access-kwlg2\") pod \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\" (UID: \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\") " Nov 03 17:07:58.959871 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:58.959745 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e624cd3f-1170-4a5f-8f14-9aa63055dce6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e624cd3f-1170-4a5f-8f14-9aa63055dce6-provider-credentials\") pod \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\" (UID: \"e624cd3f-1170-4a5f-8f14-9aa63055dce6\") " Nov 03 17:07:58.959997 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:58.959968 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e624cd3f-1170-4a5f-8f14-9aa63055dce6-routing-overlay-volume" pod "e624cd3f-1170-4a5f-8f14-9aa63055dce6" (UID: "e624cd3f-1170-4a5f-8f14-9aa63055dce6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:58.960056 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:58.960026 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e624cd3f-1170-4a5f-8f14-9aa63055dce6-plugins-config-volume" pod "e624cd3f-1170-4a5f-8f14-9aa63055dce6" (UID: "e624cd3f-1170-4a5f-8f14-9aa63055dce6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:07:58.960556 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:58.960533 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e624cd3f-1170-4a5f-8f14-9aa63055dce6-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:58.960556 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:58.960555 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e624cd3f-1170-4a5f-8f14-9aa63055dce6-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:58.962257 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:58.962234 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e624cd3f-1170-4a5f-8f14-9aa63055dce6-kube-api-access-kwlg2" pod "e624cd3f-1170-4a5f-8f14-9aa63055dce6" (UID: "e624cd3f-1170-4a5f-8f14-9aa63055dce6"). InnerVolumeSpecName "kube-api-access-kwlg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:58.962484 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:58.962456 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e624cd3f-1170-4a5f-8f14-9aa63055dce6-provider-credentials" pod "e624cd3f-1170-4a5f-8f14-9aa63055dce6" (UID: "e624cd3f-1170-4a5f-8f14-9aa63055dce6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:07:59.062645 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:59.062564 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwlg2\" (UniqueName: \"kubernetes.io/projected/e624cd3f-1170-4a5f-8f14-9aa63055dce6-kube-api-access-kwlg2\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:59.062645 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:59.062594 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e624cd3f-1170-4a5f-8f14-9aa63055dce6-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:07:59.617465 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:59.617433 4947 generic.go:356] "Generic (PLEG): container finished" podID="e624cd3f-1170-4a5f-8f14-9aa63055dce6" containerID="6cb3f8aeb445b1acc6b926ef8895f73218b383f84e233dfdf99443297e235e91" exitCode=0 Nov 03 17:07:59.617910 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:59.617484 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" Nov 03 17:07:59.617910 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:59.617522 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" event={"ID":"e624cd3f-1170-4a5f-8f14-9aa63055dce6","Type":"ContainerDied","Data":"6cb3f8aeb445b1acc6b926ef8895f73218b383f84e233dfdf99443297e235e91"} Nov 03 17:07:59.617910 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:59.617574 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg" event={"ID":"e624cd3f-1170-4a5f-8f14-9aa63055dce6","Type":"ContainerDied","Data":"cdfbed753e072f64e4bfec1803214e28917d840cd78e1c62b76f5d78b12a287e"} Nov 03 17:07:59.617910 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:59.617599 4947 scope.go:122] "RemoveContainer" containerID="6cb3f8aeb445b1acc6b926ef8895f73218b383f84e233dfdf99443297e235e91" Nov 03 17:07:59.629298 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:59.629278 4947 scope.go:122] "RemoveContainer" containerID="6cb3f8aeb445b1acc6b926ef8895f73218b383f84e233dfdf99443297e235e91" Nov 03 17:07:59.629602 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:07:59.629581 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb3f8aeb445b1acc6b926ef8895f73218b383f84e233dfdf99443297e235e91\": container with ID starting with 6cb3f8aeb445b1acc6b926ef8895f73218b383f84e233dfdf99443297e235e91 not found: ID does not exist" containerID="6cb3f8aeb445b1acc6b926ef8895f73218b383f84e233dfdf99443297e235e91" Nov 03 17:07:59.629652 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:59.629612 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb3f8aeb445b1acc6b926ef8895f73218b383f84e233dfdf99443297e235e91"} err="failed to get container status \"6cb3f8aeb445b1acc6b926ef8895f73218b383f84e233dfdf99443297e235e91\": rpc error: code = NotFound desc = could not find container \"6cb3f8aeb445b1acc6b926ef8895f73218b383f84e233dfdf99443297e235e91\": container with ID starting with 6cb3f8aeb445b1acc6b926ef8895f73218b383f84e233dfdf99443297e235e91 not found: ID does not exist" Nov 03 17:07:59.674237 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:59.674205 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg"] Nov 03 17:07:59.687357 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:07:59.687332 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4dzh5xg"] Nov 03 17:08:00.133017 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:08:00.132976 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:08:00.568089 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:00.568050 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e624cd3f-1170-4a5f-8f14-9aa63055dce6" path="/var/lib/kubelet/pods/e624cd3f-1170-4a5f-8f14-9aa63055dce6/volumes" Nov 03 17:08:07.936596 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:07.936562 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947"] Nov 03 17:08:07.937333 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:07.937318 4947 cpu_manager.go:414] "containerMap: removing container" podUID="e624cd3f-1170-4a5f-8f14-9aa63055dce6" containerName="payload-processing" Nov 03 17:08:07.937391 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:07.937335 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e624cd3f-1170-4a5f-8f14-9aa63055dce6" containerName="payload-processing" Nov 03 17:08:07.937391 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:07.937349 4947 cpu_manager.go:414] "containerMap: removing container" podUID="afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd" containerName="payload-processing" Nov 03 17:08:07.937391 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:07.937354 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd" containerName="payload-processing" Nov 03 17:08:07.937518 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:07.937499 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e624cd3f-1170-4a5f-8f14-9aa63055dce6" containerName="payload-processing" Nov 03 17:08:07.937518 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:07.937509 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd6cbf7-0712-49f1-96ae-f2fcb0bf94bd" containerName="payload-processing" Nov 03 17:08:07.942406 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:07.942381 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" Nov 03 17:08:07.980207 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:07.980176 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947"] Nov 03 17:08:08.063712 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:08.063670 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947\" (UID: \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" Nov 03 17:08:08.063942 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:08.063905 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947\" (UID: \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" Nov 03 17:08:08.064065 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:08.064044 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947\" (UID: \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" Nov 03 17:08:08.064153 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:08.064134 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4kvg\" (UniqueName: \"kubernetes.io/projected/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-kube-api-access-w4kvg\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947\" (UID: \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" Nov 03 17:08:08.166021 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:08.165983 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947\" (UID: \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" Nov 03 17:08:08.166246 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:08.166121 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947\" (UID: \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" Nov 03 17:08:08.166246 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:08.166211 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4kvg\" (UniqueName: \"kubernetes.io/projected/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-kube-api-access-w4kvg\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947\" (UID: \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" Nov 03 17:08:08.166381 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:08.166316 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947\" (UID: \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" Nov 03 17:08:08.166999 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:08.166904 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947\" (UID: \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" Nov 03 17:08:08.167123 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:08.167033 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947\" (UID: \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" Nov 03 17:08:08.168950 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:08.168928 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947\" (UID: \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" Nov 03 17:08:08.191684 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:08.191622 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4kvg\" (UniqueName: \"kubernetes.io/projected/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-kube-api-access-w4kvg\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947\" (UID: \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" Nov 03 17:08:08.257430 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:08.257395 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" Nov 03 17:08:08.579299 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:08:08.579261 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9a5770_1a3e_4b75_afc9_6836b4ed4a2b.slice/crio-0b3487d51c6ea6bc2a35536318802391d72f6029f3f27875da1ab1c60b7360bc WatchSource:0}: Error finding container 0b3487d51c6ea6bc2a35536318802391d72f6029f3f27875da1ab1c60b7360bc: Status 404 returned error can't find the container with id 0b3487d51c6ea6bc2a35536318802391d72f6029f3f27875da1ab1c60b7360bc Nov 03 17:08:08.582057 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:08.582032 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947"] Nov 03 17:08:08.722998 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:08.722960 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" event={"ID":"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b","Type":"ContainerStarted","Data":"7b80ad8f1415427e098f65b1da634e3060da27e0f52e1a543a09f47245803d27"} Nov 03 17:08:08.723200 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:08.723014 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" event={"ID":"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b","Type":"ContainerStarted","Data":"0b3487d51c6ea6bc2a35536318802391d72f6029f3f27875da1ab1c60b7360bc"} Nov 03 17:08:08.723287 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:08.723202 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" Nov 03 17:08:08.770684 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:08.770630 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" podStartSLOduration=1.770613262 podStartE2EDuration="1.770613262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:08:08.769036852 +0000 UTC m=+3232.358036824" watchObservedRunningTime="2026-11-03 17:08:08.770613262 +0000 UTC m=+3232.359613233" Nov 03 17:08:19.735405 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:19.735376 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" Nov 03 17:08:19.817207 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:19.817165 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g"] Nov 03 17:08:22.773826 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:22.773791 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:08:24.817579 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:24.817498 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" podUID="01abc6c1-836a-42bb-8f32-2f1c16d37d27" containerName="payload-processing" containerID="cri-o://1412e199693b430a60fd4ded401fb1731454add5e9c24f409f32ba5bdad6908b" gracePeriod=25 Nov 03 17:08:25.108561 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.108512 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" Nov 03 17:08:25.161543 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.161514 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01abc6c1-836a-42bb-8f32-2f1c16d37d27-provider-credentials\" (UniqueName: \"kubernetes.io/projected/01abc6c1-836a-42bb-8f32-2f1c16d37d27-provider-credentials\") pod \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\" (UID: \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\") " Nov 03 17:08:25.161794 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.161605 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01abc6c1-836a-42bb-8f32-2f1c16d37d27-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01abc6c1-836a-42bb-8f32-2f1c16d37d27-routing-overlay-volume\") pod \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\" (UID: \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\") " Nov 03 17:08:25.161794 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.161651 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01abc6c1-836a-42bb-8f32-2f1c16d37d27-kube-api-access-725vh\" (UniqueName: \"kubernetes.io/projected/01abc6c1-836a-42bb-8f32-2f1c16d37d27-kube-api-access-725vh\") pod \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\" (UID: \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\") " Nov 03 17:08:25.161963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.161804 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01abc6c1-836a-42bb-8f32-2f1c16d37d27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01abc6c1-836a-42bb-8f32-2f1c16d37d27-plugins-config-volume\") pod \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\" (UID: \"01abc6c1-836a-42bb-8f32-2f1c16d37d27\") " Nov 03 17:08:25.162182 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.162132 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01abc6c1-836a-42bb-8f32-2f1c16d37d27-routing-overlay-volume" pod "01abc6c1-836a-42bb-8f32-2f1c16d37d27" (UID: "01abc6c1-836a-42bb-8f32-2f1c16d37d27"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:08:25.162313 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.162235 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01abc6c1-836a-42bb-8f32-2f1c16d37d27-plugins-config-volume" pod "01abc6c1-836a-42bb-8f32-2f1c16d37d27" (UID: "01abc6c1-836a-42bb-8f32-2f1c16d37d27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:08:25.162632 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.162599 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/01abc6c1-836a-42bb-8f32-2f1c16d37d27-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:08:25.162632 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.162629 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01abc6c1-836a-42bb-8f32-2f1c16d37d27-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:08:25.164949 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.164924 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01abc6c1-836a-42bb-8f32-2f1c16d37d27-kube-api-access-725vh" pod "01abc6c1-836a-42bb-8f32-2f1c16d37d27" (UID: "01abc6c1-836a-42bb-8f32-2f1c16d37d27"). InnerVolumeSpecName "kube-api-access-725vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:08:25.165986 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.165945 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01abc6c1-836a-42bb-8f32-2f1c16d37d27-provider-credentials" pod "01abc6c1-836a-42bb-8f32-2f1c16d37d27" (UID: "01abc6c1-836a-42bb-8f32-2f1c16d37d27"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:08:25.263969 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.263933 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/01abc6c1-836a-42bb-8f32-2f1c16d37d27-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:08:25.263969 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.263966 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-725vh\" (UniqueName: \"kubernetes.io/projected/01abc6c1-836a-42bb-8f32-2f1c16d37d27-kube-api-access-725vh\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:08:25.884938 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.884889 4947 generic.go:356] "Generic (PLEG): container finished" podID="01abc6c1-836a-42bb-8f32-2f1c16d37d27" containerID="1412e199693b430a60fd4ded401fb1731454add5e9c24f409f32ba5bdad6908b" exitCode=0 Nov 03 17:08:25.884938 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.884937 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" event={"ID":"01abc6c1-836a-42bb-8f32-2f1c16d37d27","Type":"ContainerDied","Data":"1412e199693b430a60fd4ded401fb1731454add5e9c24f409f32ba5bdad6908b"} Nov 03 17:08:25.885469 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.884955 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" Nov 03 17:08:25.885469 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.884984 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g" event={"ID":"01abc6c1-836a-42bb-8f32-2f1c16d37d27","Type":"ContainerDied","Data":"d05f4c8dd42a6ee559b9bf404529ce43889e3c8fb115e739c1251fe715e8887a"} Nov 03 17:08:25.885469 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.885007 4947 scope.go:122] "RemoveContainer" containerID="1412e199693b430a60fd4ded401fb1731454add5e9c24f409f32ba5bdad6908b" Nov 03 17:08:25.897297 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.897276 4947 scope.go:122] "RemoveContainer" containerID="1412e199693b430a60fd4ded401fb1731454add5e9c24f409f32ba5bdad6908b" Nov 03 17:08:25.897611 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:08:25.897587 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1412e199693b430a60fd4ded401fb1731454add5e9c24f409f32ba5bdad6908b\": container with ID starting with 1412e199693b430a60fd4ded401fb1731454add5e9c24f409f32ba5bdad6908b not found: ID does not exist" containerID="1412e199693b430a60fd4ded401fb1731454add5e9c24f409f32ba5bdad6908b" Nov 03 17:08:25.897661 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.897625 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1412e199693b430a60fd4ded401fb1731454add5e9c24f409f32ba5bdad6908b"} err="failed to get container status \"1412e199693b430a60fd4ded401fb1731454add5e9c24f409f32ba5bdad6908b\": rpc error: code = NotFound desc = could not find container \"1412e199693b430a60fd4ded401fb1731454add5e9c24f409f32ba5bdad6908b\": container with ID starting with 1412e199693b430a60fd4ded401fb1731454add5e9c24f409f32ba5bdad6908b not found: ID does not exist" Nov 03 17:08:25.943102 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.943062 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g"] Nov 03 17:08:25.987724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:25.974374 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-f697ds44g"] Nov 03 17:08:26.568257 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:26.568227 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01abc6c1-836a-42bb-8f32-2f1c16d37d27" path="/var/lib/kubelet/pods/01abc6c1-836a-42bb-8f32-2f1c16d37d27/volumes" Nov 03 17:08:30.132198 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:08:30.132158 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:08:31.499198 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.499159 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl"] Nov 03 17:08:31.499979 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.499964 4947 cpu_manager.go:414] "containerMap: removing container" podUID="01abc6c1-836a-42bb-8f32-2f1c16d37d27" containerName="payload-processing" Nov 03 17:08:31.500062 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.499980 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01abc6c1-836a-42bb-8f32-2f1c16d37d27" containerName="payload-processing" Nov 03 17:08:31.500111 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.500099 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="01abc6c1-836a-42bb-8f32-2f1c16d37d27" containerName="payload-processing" Nov 03 17:08:31.505460 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.505431 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" Nov 03 17:08:31.526300 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.526264 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl"] Nov 03 17:08:31.599261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.599224 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl\" (UID: \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" Nov 03 17:08:31.599261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.599275 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl\" (UID: \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" Nov 03 17:08:31.599476 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.599383 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl\" (UID: \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" Nov 03 17:08:31.599603 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.599581 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7rpd\" (UniqueName: \"kubernetes.io/projected/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-kube-api-access-h7rpd\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl\" (UID: \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" Nov 03 17:08:31.701403 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.701368 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7rpd\" (UniqueName: \"kubernetes.io/projected/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-kube-api-access-h7rpd\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl\" (UID: \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" Nov 03 17:08:31.701589 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.701489 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl\" (UID: \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" Nov 03 17:08:31.701589 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.701518 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl\" (UID: \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" Nov 03 17:08:31.701589 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.701581 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl\" (UID: \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" Nov 03 17:08:31.702228 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.702195 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl\" (UID: \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" Nov 03 17:08:31.702375 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.702352 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl\" (UID: \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" Nov 03 17:08:31.704340 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.704314 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl\" (UID: \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" Nov 03 17:08:31.727165 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.727106 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7rpd\" (UniqueName: \"kubernetes.io/projected/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-kube-api-access-h7rpd\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl\" (UID: \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" Nov 03 17:08:31.823872 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:31.823777 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" Nov 03 17:08:32.135590 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:08:32.135553 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101a5bb6_0ff8_4498_82d1_c75ddbac14a1.slice/crio-b25cdb46c422f192bd48a932c0491b94602910331b98a78301c2c2ee7e23cab2 WatchSource:0}: Error finding container b25cdb46c422f192bd48a932c0491b94602910331b98a78301c2c2ee7e23cab2: Status 404 returned error can't find the container with id b25cdb46c422f192bd48a932c0491b94602910331b98a78301c2c2ee7e23cab2 Nov 03 17:08:32.136167 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:32.136144 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl"] Nov 03 17:08:32.948612 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:32.948574 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" event={"ID":"101a5bb6-0ff8-4498-82d1-c75ddbac14a1","Type":"ContainerStarted","Data":"c9d52a341a327253ebdae7156b10714d462515015e31114c88202f758fa12eba"} Nov 03 17:08:32.949109 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:32.948678 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" event={"ID":"101a5bb6-0ff8-4498-82d1-c75ddbac14a1","Type":"ContainerStarted","Data":"b25cdb46c422f192bd48a932c0491b94602910331b98a78301c2c2ee7e23cab2"} Nov 03 17:08:32.949109 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:32.948866 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" Nov 03 17:08:32.996491 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:32.996438 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" podStartSLOduration=1.996425145 podStartE2EDuration="1.996425145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:08:32.993079506 +0000 UTC m=+3256.582079476" watchObservedRunningTime="2026-11-03 17:08:32.996425145 +0000 UTC m=+3256.585425113" Nov 03 17:08:43.960969 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:43.960922 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" Nov 03 17:08:44.046893 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:44.046858 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947"] Nov 03 17:08:49.048186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:49.048141 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" podUID="9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b" containerName="payload-processing" containerID="cri-o://7b80ad8f1415427e098f65b1da634e3060da27e0f52e1a543a09f47245803d27" gracePeriod=25 Nov 03 17:08:49.396125 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:49.396096 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" Nov 03 17:08:49.448926 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:49.448887 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-routing-overlay-volume\") pod \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\" (UID: \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\") " Nov 03 17:08:49.449147 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:49.448995 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-provider-credentials\") pod \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\" (UID: \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\") " Nov 03 17:08:49.449217 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:49.449155 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-plugins-config-volume\") pod \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\" (UID: \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\") " Nov 03 17:08:49.449217 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:49.449191 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-kube-api-access-w4kvg\" (UniqueName: \"kubernetes.io/projected/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-kube-api-access-w4kvg\") pod \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\" (UID: \"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b\") " Nov 03 17:08:49.450113 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:49.449652 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-routing-overlay-volume" pod "9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b" (UID: "9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:08:49.450113 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:49.450038 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-plugins-config-volume" pod "9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b" (UID: "9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:08:49.450365 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:49.450155 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:08:49.450365 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:49.450179 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:08:49.470254 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:49.470218 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-provider-credentials" pod "9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b" (UID: "9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:08:49.470421 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:49.470272 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-kube-api-access-w4kvg" pod "9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b" (UID: "9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b"). InnerVolumeSpecName "kube-api-access-w4kvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:08:49.551475 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:49.551437 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4kvg\" (UniqueName: \"kubernetes.io/projected/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-kube-api-access-w4kvg\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:08:49.551475 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:49.551469 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:08:50.130661 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:50.130622 4947 generic.go:356] "Generic (PLEG): container finished" podID="9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b" containerID="7b80ad8f1415427e098f65b1da634e3060da27e0f52e1a543a09f47245803d27" exitCode=0 Nov 03 17:08:50.131161 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:50.130671 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" Nov 03 17:08:50.131161 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:50.130730 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" event={"ID":"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b","Type":"ContainerDied","Data":"7b80ad8f1415427e098f65b1da634e3060da27e0f52e1a543a09f47245803d27"} Nov 03 17:08:50.131161 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:50.130781 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947" event={"ID":"9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b","Type":"ContainerDied","Data":"0b3487d51c6ea6bc2a35536318802391d72f6029f3f27875da1ab1c60b7360bc"} Nov 03 17:08:50.131161 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:50.130803 4947 scope.go:122] "RemoveContainer" containerID="7b80ad8f1415427e098f65b1da634e3060da27e0f52e1a543a09f47245803d27" Nov 03 17:08:50.142849 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:50.142828 4947 scope.go:122] "RemoveContainer" containerID="7b80ad8f1415427e098f65b1da634e3060da27e0f52e1a543a09f47245803d27" Nov 03 17:08:50.143164 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:08:50.143144 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b80ad8f1415427e098f65b1da634e3060da27e0f52e1a543a09f47245803d27\": container with ID starting with 7b80ad8f1415427e098f65b1da634e3060da27e0f52e1a543a09f47245803d27 not found: ID does not exist" containerID="7b80ad8f1415427e098f65b1da634e3060da27e0f52e1a543a09f47245803d27" Nov 03 17:08:50.143215 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:50.143172 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b80ad8f1415427e098f65b1da634e3060da27e0f52e1a543a09f47245803d27"} err="failed to get container status \"7b80ad8f1415427e098f65b1da634e3060da27e0f52e1a543a09f47245803d27\": rpc error: code = NotFound desc = could not find container \"7b80ad8f1415427e098f65b1da634e3060da27e0f52e1a543a09f47245803d27\": container with ID starting with 7b80ad8f1415427e098f65b1da634e3060da27e0f52e1a543a09f47245803d27 not found: ID does not exist" Nov 03 17:08:50.189751 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:50.189713 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947"] Nov 03 17:08:50.201444 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:50.201414 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-58b6tk947"] Nov 03 17:08:50.568166 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:50.568131 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b" path="/var/lib/kubelet/pods/9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b/volumes" Nov 03 17:08:54.888531 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:54.888487 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh"] Nov 03 17:08:54.890465 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:54.890437 4947 cpu_manager.go:414] "containerMap: removing container" podUID="9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b" containerName="payload-processing" Nov 03 17:08:54.890643 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:54.890627 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b" containerName="payload-processing" Nov 03 17:08:54.894168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:54.894148 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c9a5770-1a3e-4b75-afc9-6836b4ed4a2b" containerName="payload-processing" Nov 03 17:08:54.905217 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:54.905191 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" Nov 03 17:08:54.952983 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:54.940447 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh"] Nov 03 17:08:55.069977 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:55.069939 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh\" (UID: \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" Nov 03 17:08:55.070187 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:55.070173 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh\" (UID: \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" Nov 03 17:08:55.070333 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:55.070305 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpg2z\" (UniqueName: \"kubernetes.io/projected/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-kube-api-access-tpg2z\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh\" (UID: \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" Nov 03 17:08:55.070551 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:55.070500 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh\" (UID: \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" Nov 03 17:08:55.171808 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:55.171689 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh\" (UID: \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" Nov 03 17:08:55.171996 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:55.171828 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpg2z\" (UniqueName: \"kubernetes.io/projected/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-kube-api-access-tpg2z\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh\" (UID: \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" Nov 03 17:08:55.171996 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:55.171923 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh\" (UID: \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" Nov 03 17:08:55.171996 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:55.171942 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh\" (UID: \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" Nov 03 17:08:55.172489 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:55.172463 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh\" (UID: \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" Nov 03 17:08:55.172597 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:55.172579 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh\" (UID: \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" Nov 03 17:08:55.174804 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:55.174780 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh\" (UID: \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" Nov 03 17:08:55.196607 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:55.196582 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpg2z\" (UniqueName: \"kubernetes.io/projected/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-kube-api-access-tpg2z\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh\" (UID: \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" Nov 03 17:08:55.302065 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:55.302036 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" Nov 03 17:08:55.618384 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:08:55.618328 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbeaf8b_c161_4a9d_91ab_eeed04ac03fa.slice/crio-c7c06faf64bf603b072df0903b5d2fb4c9df244fab2342785f47590e104b51b2 WatchSource:0}: Error finding container c7c06faf64bf603b072df0903b5d2fb4c9df244fab2342785f47590e104b51b2: Status 404 returned error can't find the container with id c7c06faf64bf603b072df0903b5d2fb4c9df244fab2342785f47590e104b51b2 Nov 03 17:08:55.618986 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:55.618957 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh"] Nov 03 17:08:56.221779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:56.218132 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" event={"ID":"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa","Type":"ContainerStarted","Data":"1836591c247f58ea8d0e452f91d0883ebcbab6bfc5fb096d4dbe66e7cab06aa2"} Nov 03 17:08:56.221779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:56.218234 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" event={"ID":"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa","Type":"ContainerStarted","Data":"c7c06faf64bf603b072df0903b5d2fb4c9df244fab2342785f47590e104b51b2"} Nov 03 17:08:56.221779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:56.219279 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" Nov 03 17:08:56.276783 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:08:56.271456 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" podStartSLOduration=2.271437236 podStartE2EDuration="2.271437236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:08:56.270153607 +0000 UTC m=+3279.859153579" watchObservedRunningTime="2026-11-03 17:08:56.271437236 +0000 UTC m=+3279.860437209" Nov 03 17:09:00.159803 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:00.159116 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:09:01.316552 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:01.316510 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl"] Nov 03 17:09:01.322306 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:01.322277 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" Nov 03 17:09:01.342260 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:01.342227 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl"] Nov 03 17:09:01.343687 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:01.343662 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a0e27\"/\"agc-cred-d025f4746e15\"" Nov 03 17:09:01.424324 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:01.424280 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e386c866-ccc7-4395-8866-b4650f18eb5e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl\" (UID: \"e386c866-ccc7-4395-8866-b4650f18eb5e\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" Nov 03 17:09:01.424532 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:01.424344 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69jhg\" (UniqueName: \"kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-kube-api-access-69jhg\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl\" (UID: \"e386c866-ccc7-4395-8866-b4650f18eb5e\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" Nov 03 17:09:01.424585 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:01.424533 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl\" (UID: \"e386c866-ccc7-4395-8866-b4650f18eb5e\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" Nov 03 17:09:01.424789 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:01.424765 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e386c866-ccc7-4395-8866-b4650f18eb5e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl\" (UID: \"e386c866-ccc7-4395-8866-b4650f18eb5e\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" Nov 03 17:09:01.525917 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:01.525880 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e386c866-ccc7-4395-8866-b4650f18eb5e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl\" (UID: \"e386c866-ccc7-4395-8866-b4650f18eb5e\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" Nov 03 17:09:01.526119 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:01.525929 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69jhg\" (UniqueName: \"kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-kube-api-access-69jhg\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl\" (UID: \"e386c866-ccc7-4395-8866-b4650f18eb5e\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" Nov 03 17:09:01.526119 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:01.526010 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl\" (UID: \"e386c866-ccc7-4395-8866-b4650f18eb5e\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" Nov 03 17:09:01.526253 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:01.526135 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e386c866-ccc7-4395-8866-b4650f18eb5e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl\" (UID: \"e386c866-ccc7-4395-8866-b4650f18eb5e\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" Nov 03 17:09:01.526722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:01.526671 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e386c866-ccc7-4395-8866-b4650f18eb5e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl\" (UID: \"e386c866-ccc7-4395-8866-b4650f18eb5e\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" Nov 03 17:09:01.526821 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:01.526679 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e386c866-ccc7-4395-8866-b4650f18eb5e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl\" (UID: \"e386c866-ccc7-4395-8866-b4650f18eb5e\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" Nov 03 17:09:01.528919 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:01.528886 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl\" (UID: \"e386c866-ccc7-4395-8866-b4650f18eb5e\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" Nov 03 17:09:01.551447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:01.551406 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69jhg\" (UniqueName: \"kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-kube-api-access-69jhg\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl\" (UID: \"e386c866-ccc7-4395-8866-b4650f18eb5e\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" Nov 03 17:09:01.643759 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:01.643655 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" Nov 03 17:09:01.981548 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:01.981446 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl"] Nov 03 17:09:01.998343 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:09:01.998306 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode386c866_ccc7_4395_8866_b4650f18eb5e.slice/crio-633810d179ca05c62c1da7e86bb571c9a0572c0b8209a363a9429e54473271f4 WatchSource:0}: Error finding container 633810d179ca05c62c1da7e86bb571c9a0572c0b8209a363a9429e54473271f4: Status 404 returned error can't find the container with id 633810d179ca05c62c1da7e86bb571c9a0572c0b8209a363a9429e54473271f4 Nov 03 17:09:02.306491 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:02.306446 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" event={"ID":"e386c866-ccc7-4395-8866-b4650f18eb5e","Type":"ContainerStarted","Data":"c1464655ccef48e4a2ca974d7a969dbfecad7224ba8d01bdbcb62014b4ef07d2"} Nov 03 17:09:02.306710 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:02.306529 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" event={"ID":"e386c866-ccc7-4395-8866-b4650f18eb5e","Type":"ContainerStarted","Data":"633810d179ca05c62c1da7e86bb571c9a0572c0b8209a363a9429e54473271f4"} Nov 03 17:09:02.306710 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:02.306573 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" Nov 03 17:09:02.359596 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:02.359539 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" podStartSLOduration=1.359524406 podStartE2EDuration="1.359524406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:09:02.355564065 +0000 UTC m=+3285.944564034" watchObservedRunningTime="2026-11-03 17:09:02.359524406 +0000 UTC m=+3285.948524374" Nov 03 17:09:08.236266 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:08.236232 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" Nov 03 17:09:08.319396 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:08.319355 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v"] Nov 03 17:09:12.678595 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:12.678560 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl"] Nov 03 17:09:13.320421 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:13.320379 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" podUID="ee041d48-e7d3-4666-97a3-bd9487b18de9" containerName="payload-processing" containerID="cri-o://f5ae128ad904356940fed5e8bea2a49f2b7296855d81f3450ef1db13ffa34fff" gracePeriod=25 Nov 03 17:09:13.367948 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:13.367915 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" Nov 03 17:09:13.548234 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:13.548199 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j"] Nov 03 17:09:13.885532 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:13.885502 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" Nov 03 17:09:14.005597 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.005561 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee041d48-e7d3-4666-97a3-bd9487b18de9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee041d48-e7d3-4666-97a3-bd9487b18de9-routing-overlay-volume\") pod \"ee041d48-e7d3-4666-97a3-bd9487b18de9\" (UID: \"ee041d48-e7d3-4666-97a3-bd9487b18de9\") " Nov 03 17:09:14.005816 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.005791 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee041d48-e7d3-4666-97a3-bd9487b18de9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee041d48-e7d3-4666-97a3-bd9487b18de9-provider-credentials\") pod \"ee041d48-e7d3-4666-97a3-bd9487b18de9\" (UID: \"ee041d48-e7d3-4666-97a3-bd9487b18de9\") " Nov 03 17:09:14.006638 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.006405 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee041d48-e7d3-4666-97a3-bd9487b18de9-routing-overlay-volume" pod "ee041d48-e7d3-4666-97a3-bd9487b18de9" (UID: "ee041d48-e7d3-4666-97a3-bd9487b18de9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:14.006638 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.006563 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee041d48-e7d3-4666-97a3-bd9487b18de9-kube-api-access-h9xjw\" (UniqueName: \"kubernetes.io/projected/ee041d48-e7d3-4666-97a3-bd9487b18de9-kube-api-access-h9xjw\") pod \"ee041d48-e7d3-4666-97a3-bd9487b18de9\" (UID: \"ee041d48-e7d3-4666-97a3-bd9487b18de9\") " Nov 03 17:09:14.006638 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.006607 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee041d48-e7d3-4666-97a3-bd9487b18de9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee041d48-e7d3-4666-97a3-bd9487b18de9-plugins-config-volume\") pod \"ee041d48-e7d3-4666-97a3-bd9487b18de9\" (UID: \"ee041d48-e7d3-4666-97a3-bd9487b18de9\") " Nov 03 17:09:14.008391 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.008362 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ee041d48-e7d3-4666-97a3-bd9487b18de9-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:14.008884 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.008859 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee041d48-e7d3-4666-97a3-bd9487b18de9-plugins-config-volume" pod "ee041d48-e7d3-4666-97a3-bd9487b18de9" (UID: "ee041d48-e7d3-4666-97a3-bd9487b18de9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:14.011099 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.011069 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee041d48-e7d3-4666-97a3-bd9487b18de9-kube-api-access-h9xjw" pod "ee041d48-e7d3-4666-97a3-bd9487b18de9" (UID: "ee041d48-e7d3-4666-97a3-bd9487b18de9"). InnerVolumeSpecName "kube-api-access-h9xjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:14.011957 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.011931 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee041d48-e7d3-4666-97a3-bd9487b18de9-provider-credentials" pod "ee041d48-e7d3-4666-97a3-bd9487b18de9" (UID: "ee041d48-e7d3-4666-97a3-bd9487b18de9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:14.112752 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.112691 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9xjw\" (UniqueName: \"kubernetes.io/projected/ee041d48-e7d3-4666-97a3-bd9487b18de9-kube-api-access-h9xjw\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:14.112999 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.112979 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee041d48-e7d3-4666-97a3-bd9487b18de9-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:14.127725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.113116 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ee041d48-e7d3-4666-97a3-bd9487b18de9-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:14.501441 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.501408 4947 generic.go:356] "Generic (PLEG): container finished" podID="ee041d48-e7d3-4666-97a3-bd9487b18de9" containerID="f5ae128ad904356940fed5e8bea2a49f2b7296855d81f3450ef1db13ffa34fff" exitCode=0 Nov 03 17:09:14.501643 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.501459 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" Nov 03 17:09:14.501643 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.501470 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" event={"ID":"ee041d48-e7d3-4666-97a3-bd9487b18de9","Type":"ContainerDied","Data":"f5ae128ad904356940fed5e8bea2a49f2b7296855d81f3450ef1db13ffa34fff"} Nov 03 17:09:14.501643 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.501525 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v" event={"ID":"ee041d48-e7d3-4666-97a3-bd9487b18de9","Type":"ContainerDied","Data":"4cd560b143eb241114472cbf580a358a6c1a9d755710fcc2a577012e40e7989f"} Nov 03 17:09:14.501643 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.501548 4947 scope.go:122] "RemoveContainer" containerID="f5ae128ad904356940fed5e8bea2a49f2b7296855d81f3450ef1db13ffa34fff" Nov 03 17:09:14.513861 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.513838 4947 scope.go:122] "RemoveContainer" containerID="f5ae128ad904356940fed5e8bea2a49f2b7296855d81f3450ef1db13ffa34fff" Nov 03 17:09:14.514210 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:14.514188 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5ae128ad904356940fed5e8bea2a49f2b7296855d81f3450ef1db13ffa34fff\": container with ID starting with f5ae128ad904356940fed5e8bea2a49f2b7296855d81f3450ef1db13ffa34fff not found: ID does not exist" containerID="f5ae128ad904356940fed5e8bea2a49f2b7296855d81f3450ef1db13ffa34fff" Nov 03 17:09:14.514285 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.514219 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5ae128ad904356940fed5e8bea2a49f2b7296855d81f3450ef1db13ffa34fff"} err="failed to get container status \"f5ae128ad904356940fed5e8bea2a49f2b7296855d81f3450ef1db13ffa34fff\": rpc error: code = NotFound desc = could not find container \"f5ae128ad904356940fed5e8bea2a49f2b7296855d81f3450ef1db13ffa34fff\": container with ID starting with f5ae128ad904356940fed5e8bea2a49f2b7296855d81f3450ef1db13ffa34fff not found: ID does not exist" Nov 03 17:09:14.781568 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.781480 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v"] Nov 03 17:09:14.781568 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.781544 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7d96v277v"] Nov 03 17:09:14.781917 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.781574 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x"] Nov 03 17:09:14.782847 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.782820 4947 cpu_manager.go:414] "containerMap: removing container" podUID="ee041d48-e7d3-4666-97a3-bd9487b18de9" containerName="payload-processing" Nov 03 17:09:14.782956 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.782848 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee041d48-e7d3-4666-97a3-bd9487b18de9" containerName="payload-processing" Nov 03 17:09:14.783112 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.783096 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee041d48-e7d3-4666-97a3-bd9487b18de9" containerName="payload-processing" Nov 03 17:09:14.793729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.791754 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x"] Nov 03 17:09:14.793729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.791870 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" Nov 03 17:09:14.808252 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.808223 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"payload-processing-external-model-e2e--6d96ad03-dockercfg-xfd8c\"" Nov 03 17:09:14.810549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.810513 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"agc-cred-4fb82f7c0e56\"" Nov 03 17:09:14.898022 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.897954 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x\" (UID: \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" Nov 03 17:09:14.898577 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.898237 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr6b4\" (UniqueName: \"kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-kube-api-access-zr6b4\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x\" (UID: \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" Nov 03 17:09:14.898577 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.898370 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x\" (UID: \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" Nov 03 17:09:14.898577 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:14.898422 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x\" (UID: \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" Nov 03 17:09:15.006730 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.004952 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr6b4\" (UniqueName: \"kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-kube-api-access-zr6b4\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x\" (UID: \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" Nov 03 17:09:15.006730 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.005088 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x\" (UID: \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" Nov 03 17:09:15.006730 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.005134 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x\" (UID: \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" Nov 03 17:09:15.006730 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.006397 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x\" (UID: \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" Nov 03 17:09:15.010724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.009622 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x\" (UID: \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" Nov 03 17:09:15.010724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.009777 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x\" (UID: \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" Nov 03 17:09:15.017733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.017672 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x\" (UID: \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" Nov 03 17:09:15.068779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.065482 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr6b4\" (UniqueName: \"kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-kube-api-access-zr6b4\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x\" (UID: \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" Nov 03 17:09:15.125130 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.120978 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" Nov 03 17:09:15.651079 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:09:15.651046 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf509afa_08d4_485f_8fbc_dc80c0bbec6e.slice/crio-b1053743cdd28679db8d78170204b159fd86fd620afff1ef3f408f8436cdbf80 WatchSource:0}: Error finding container b1053743cdd28679db8d78170204b159fd86fd620afff1ef3f408f8436cdbf80: Status 404 returned error can't find the container with id b1053743cdd28679db8d78170204b159fd86fd620afff1ef3f408f8436cdbf80 Nov 03 17:09:15.651365 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.651345 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x"] Nov 03 17:09:15.744839 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.744804 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw"] Nov 03 17:09:15.749521 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.749499 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" Nov 03 17:09:15.776410 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.776375 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw"] Nov 03 17:09:15.846350 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.846307 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8360f74a-698b-4eab-880f-e399f4ee23d7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw\" (UID: \"8360f74a-698b-4eab-880f-e399f4ee23d7\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" Nov 03 17:09:15.846542 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.846359 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8360f74a-698b-4eab-880f-e399f4ee23d7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw\" (UID: \"8360f74a-698b-4eab-880f-e399f4ee23d7\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" Nov 03 17:09:15.846542 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.846529 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ph4c\" (UniqueName: \"kubernetes.io/projected/8360f74a-698b-4eab-880f-e399f4ee23d7-kube-api-access-4ph4c\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw\" (UID: \"8360f74a-698b-4eab-880f-e399f4ee23d7\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" Nov 03 17:09:15.846646 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.846571 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8360f74a-698b-4eab-880f-e399f4ee23d7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw\" (UID: \"8360f74a-698b-4eab-880f-e399f4ee23d7\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" Nov 03 17:09:15.972120 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.966118 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8360f74a-698b-4eab-880f-e399f4ee23d7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw\" (UID: \"8360f74a-698b-4eab-880f-e399f4ee23d7\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" Nov 03 17:09:15.972120 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.966186 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8360f74a-698b-4eab-880f-e399f4ee23d7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw\" (UID: \"8360f74a-698b-4eab-880f-e399f4ee23d7\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" Nov 03 17:09:15.972120 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.966357 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ph4c\" (UniqueName: \"kubernetes.io/projected/8360f74a-698b-4eab-880f-e399f4ee23d7-kube-api-access-4ph4c\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw\" (UID: \"8360f74a-698b-4eab-880f-e399f4ee23d7\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" Nov 03 17:09:15.972120 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.966407 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8360f74a-698b-4eab-880f-e399f4ee23d7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw\" (UID: \"8360f74a-698b-4eab-880f-e399f4ee23d7\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" Nov 03 17:09:15.996721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.982294 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8360f74a-698b-4eab-880f-e399f4ee23d7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw\" (UID: \"8360f74a-698b-4eab-880f-e399f4ee23d7\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" Nov 03 17:09:15.996721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.983058 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8360f74a-698b-4eab-880f-e399f4ee23d7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw\" (UID: \"8360f74a-698b-4eab-880f-e399f4ee23d7\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" Nov 03 17:09:15.996721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:15.985364 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8360f74a-698b-4eab-880f-e399f4ee23d7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw\" (UID: \"8360f74a-698b-4eab-880f-e399f4ee23d7\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" Nov 03 17:09:16.012865 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:16.012828 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ph4c\" (UniqueName: \"kubernetes.io/projected/8360f74a-698b-4eab-880f-e399f4ee23d7-kube-api-access-4ph4c\") pod \"payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw\" (UID: \"8360f74a-698b-4eab-880f-e399f4ee23d7\") " pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" Nov 03 17:09:16.064017 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:16.063984 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" Nov 03 17:09:16.284021 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:16.283984 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x"] Nov 03 17:09:16.385547 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:09:16.385510 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8360f74a_698b_4eab_880f_e399f4ee23d7.slice/crio-04a215f6a5a998fc57f545577fed983ca745b9aa25a5b57c815a8f178c078f71 WatchSource:0}: Error finding container 04a215f6a5a998fc57f545577fed983ca745b9aa25a5b57c815a8f178c078f71: Status 404 returned error can't find the container with id 04a215f6a5a998fc57f545577fed983ca745b9aa25a5b57c815a8f178c078f71 Nov 03 17:09:16.388970 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:16.388944 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw"] Nov 03 17:09:16.416103 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:16.416073 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:09:16.568286 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:16.568200 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee041d48-e7d3-4666-97a3-bd9487b18de9" path="/var/lib/kubelet/pods/ee041d48-e7d3-4666-97a3-bd9487b18de9/volumes" Nov 03 17:09:16.573603 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:16.573554 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" event={"ID":"8360f74a-698b-4eab-880f-e399f4ee23d7","Type":"ContainerStarted","Data":"2e7cc9d908b990a3c0909bf053a563b0318f727ee4b50c0187d5d4c70de5ebf1"} Nov 03 17:09:16.573824 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:16.573614 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" event={"ID":"8360f74a-698b-4eab-880f-e399f4ee23d7","Type":"ContainerStarted","Data":"04a215f6a5a998fc57f545577fed983ca745b9aa25a5b57c815a8f178c078f71"} Nov 03 17:09:16.573824 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:16.573646 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" Nov 03 17:09:16.575439 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:16.575413 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" event={"ID":"df509afa-08d4-485f-8fbc-dc80c0bbec6e","Type":"ContainerStarted","Data":"5894154601e062072aa5b0bbb052afebe47211f3a9639a848c20237b2dc6af31"} Nov 03 17:09:16.575544 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:16.575459 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" event={"ID":"df509afa-08d4-485f-8fbc-dc80c0bbec6e","Type":"ContainerStarted","Data":"b1053743cdd28679db8d78170204b159fd86fd620afff1ef3f408f8436cdbf80"} Nov 03 17:09:16.575544 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:16.575481 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" Nov 03 17:09:16.575615 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:16.575552 4947 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" secret="" err="secret \"payload-processing-external-model-e2e--6d96ad03-dockercfg-xfd8c\" not found" Nov 03 17:09:16.695742 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:16.687516 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2492c4/agc-cred-4fb82f7c0e56: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:09:16.695742 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:16.687544 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:09:16.695742 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:16.687598 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-provider-credentials podName:df509afa-08d4-485f-8fbc-dc80c0bbec6e nodeName:}" failed. No retries permitted until 2026-11-03 17:09:17.18757924 +0000 UTC m=+3300.776579201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" (UID: "df509afa-08d4-485f-8fbc-dc80c0bbec6e") : secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:09:16.706724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:16.696987 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" podStartSLOduration=1.6969652769999999 podStartE2EDuration="1.696965277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:09:16.622851165 +0000 UTC m=+3300.211851137" watchObservedRunningTime="2026-11-03 17:09:16.696965277 +0000 UTC m=+3300.285965250" Nov 03 17:09:16.706724 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:16.699255 4947 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-plugins-e2e-extmod-a-2492c4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-2492c4" not found Nov 03 17:09:16.706724 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:16.699326 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-plugins-config-volume podName:df509afa-08d4-485f-8fbc-dc80c0bbec6e nodeName:}" failed. No retries permitted until 2026-11-03 17:09:17.199307399 +0000 UTC m=+3300.788307360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" (UID: "df509afa-08d4-485f-8fbc-dc80c0bbec6e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-2492c4" not found Nov 03 17:09:16.765720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:16.762277 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" podStartSLOduration=2.7622419049999998 podStartE2EDuration="2.762241905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:09:16.702775212 +0000 UTC m=+3300.291775183" watchObservedRunningTime="2026-11-03 17:09:16.762241905 +0000 UTC m=+3300.351241875" Nov 03 17:09:17.066670 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:17.066631 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:09:17.069590 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:17.069567 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:09:17.195929 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:17.195892 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2492c4/agc-cred-4fb82f7c0e56: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:09:17.195929 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:17.195920 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:09:17.196193 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:17.195980 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-provider-credentials podName:df509afa-08d4-485f-8fbc-dc80c0bbec6e nodeName:}" failed. No retries permitted until 2026-11-03 17:09:18.1959661 +0000 UTC m=+3301.784966048 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" (UID: "df509afa-08d4-485f-8fbc-dc80c0bbec6e") : secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:09:17.296949 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:17.296907 4947 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-plugins-e2e-extmod-a-2492c4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-2492c4" not found Nov 03 17:09:17.297144 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:17.296988 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-plugins-config-volume podName:df509afa-08d4-485f-8fbc-dc80c0bbec6e nodeName:}" failed. No retries permitted until 2026-11-03 17:09:18.296972503 +0000 UTC m=+3301.885972467 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" (UID: "df509afa-08d4-485f-8fbc-dc80c0bbec6e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-2492c4" not found Nov 03 17:09:17.679054 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:17.678991 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" podUID="d4cda53a-2726-4dc0-8057-f3302901c980" containerName="payload-processing" containerID="cri-o://3db109b67775216034362248e9f2cbbbe3e3db20e4a78582099f7cafe1f1a413" gracePeriod=25 Nov 03 17:09:18.010000 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.009967 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" Nov 03 17:09:18.029490 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.029456 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4cda53a-2726-4dc0-8057-f3302901c980-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4cda53a-2726-4dc0-8057-f3302901c980-routing-overlay-volume\") pod \"d4cda53a-2726-4dc0-8057-f3302901c980\" (UID: \"d4cda53a-2726-4dc0-8057-f3302901c980\") " Nov 03 17:09:18.029749 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.029537 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4cda53a-2726-4dc0-8057-f3302901c980-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4cda53a-2726-4dc0-8057-f3302901c980-provider-credentials\") pod \"d4cda53a-2726-4dc0-8057-f3302901c980\" (UID: \"d4cda53a-2726-4dc0-8057-f3302901c980\") " Nov 03 17:09:18.029749 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.029615 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4cda53a-2726-4dc0-8057-f3302901c980-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cda53a-2726-4dc0-8057-f3302901c980-plugins-config-volume\") pod \"d4cda53a-2726-4dc0-8057-f3302901c980\" (UID: \"d4cda53a-2726-4dc0-8057-f3302901c980\") " Nov 03 17:09:18.029907 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.029773 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4cda53a-2726-4dc0-8057-f3302901c980-kube-api-access-lpflr\" (UniqueName: \"kubernetes.io/projected/d4cda53a-2726-4dc0-8057-f3302901c980-kube-api-access-lpflr\") pod \"d4cda53a-2726-4dc0-8057-f3302901c980\" (UID: \"d4cda53a-2726-4dc0-8057-f3302901c980\") " Nov 03 17:09:18.029988 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.029950 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4cda53a-2726-4dc0-8057-f3302901c980-routing-overlay-volume" pod "d4cda53a-2726-4dc0-8057-f3302901c980" (UID: "d4cda53a-2726-4dc0-8057-f3302901c980"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:18.030050 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.030026 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4cda53a-2726-4dc0-8057-f3302901c980-plugins-config-volume" pod "d4cda53a-2726-4dc0-8057-f3302901c980" (UID: "d4cda53a-2726-4dc0-8057-f3302901c980"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:18.030408 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.030384 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d4cda53a-2726-4dc0-8057-f3302901c980-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:18.030408 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.030406 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cda53a-2726-4dc0-8057-f3302901c980-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:18.032110 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.032087 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4cda53a-2726-4dc0-8057-f3302901c980-provider-credentials" pod "d4cda53a-2726-4dc0-8057-f3302901c980" (UID: "d4cda53a-2726-4dc0-8057-f3302901c980"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:18.032217 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.032146 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4cda53a-2726-4dc0-8057-f3302901c980-kube-api-access-lpflr" pod "d4cda53a-2726-4dc0-8057-f3302901c980" (UID: "d4cda53a-2726-4dc0-8057-f3302901c980"). InnerVolumeSpecName "kube-api-access-lpflr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:18.132315 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.132276 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpflr\" (UniqueName: \"kubernetes.io/projected/d4cda53a-2726-4dc0-8057-f3302901c980-kube-api-access-lpflr\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:18.132315 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.132307 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d4cda53a-2726-4dc0-8057-f3302901c980-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:18.234624 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:18.234518 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2492c4/agc-cred-4fb82f7c0e56: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:09:18.234624 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:18.234545 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:09:18.234624 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:18.234614 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-provider-credentials podName:df509afa-08d4-485f-8fbc-dc80c0bbec6e nodeName:}" failed. No retries permitted until 2026-11-03 17:09:20.234594735 +0000 UTC m=+3303.823594682 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" (UID: "df509afa-08d4-485f-8fbc-dc80c0bbec6e") : secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:09:18.335768 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:18.335735 4947 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-plugins-e2e-extmod-a-2492c4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-2492c4" not found Nov 03 17:09:18.335945 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:18.335825 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-plugins-config-volume podName:df509afa-08d4-485f-8fbc-dc80c0bbec6e nodeName:}" failed. No retries permitted until 2026-11-03 17:09:20.33580513 +0000 UTC m=+3303.924805092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" (UID: "df509afa-08d4-485f-8fbc-dc80c0bbec6e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-2492c4" not found Nov 03 17:09:18.548852 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.548813 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" podUID="2f011e68-26d2-40c9-b439-550813b3fdc9" containerName="payload-processing" containerID="cri-o://858b7a5f5da2ef0c107e6228973bc37bed51c2c13f0f72b9202d2fda1b1d2350" gracePeriod=25 Nov 03 17:09:18.621782 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.621737 4947 generic.go:356] "Generic (PLEG): container finished" podID="d4cda53a-2726-4dc0-8057-f3302901c980" containerID="3db109b67775216034362248e9f2cbbbe3e3db20e4a78582099f7cafe1f1a413" exitCode=0 Nov 03 17:09:18.621961 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.621782 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" Nov 03 17:09:18.621961 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.621818 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" event={"ID":"d4cda53a-2726-4dc0-8057-f3302901c980","Type":"ContainerDied","Data":"3db109b67775216034362248e9f2cbbbe3e3db20e4a78582099f7cafe1f1a413"} Nov 03 17:09:18.621961 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.621875 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl" event={"ID":"d4cda53a-2726-4dc0-8057-f3302901c980","Type":"ContainerDied","Data":"c58c44f2b768147e52443f2ef2e9daabc01009348c15aec18f01efd357735117"} Nov 03 17:09:18.621961 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.621900 4947 scope.go:122] "RemoveContainer" containerID="3db109b67775216034362248e9f2cbbbe3e3db20e4a78582099f7cafe1f1a413" Nov 03 17:09:18.662897 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.662872 4947 scope.go:122] "RemoveContainer" containerID="3db109b67775216034362248e9f2cbbbe3e3db20e4a78582099f7cafe1f1a413" Nov 03 17:09:18.663343 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:18.663317 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db109b67775216034362248e9f2cbbbe3e3db20e4a78582099f7cafe1f1a413\": container with ID starting with 3db109b67775216034362248e9f2cbbbe3e3db20e4a78582099f7cafe1f1a413 not found: ID does not exist" containerID="3db109b67775216034362248e9f2cbbbe3e3db20e4a78582099f7cafe1f1a413" Nov 03 17:09:18.663481 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.663352 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db109b67775216034362248e9f2cbbbe3e3db20e4a78582099f7cafe1f1a413"} err="failed to get container status \"3db109b67775216034362248e9f2cbbbe3e3db20e4a78582099f7cafe1f1a413\": rpc error: code = NotFound desc = could not find container \"3db109b67775216034362248e9f2cbbbe3e3db20e4a78582099f7cafe1f1a413\": container with ID starting with 3db109b67775216034362248e9f2cbbbe3e3db20e4a78582099f7cafe1f1a413 not found: ID does not exist" Nov 03 17:09:18.692164 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.680924 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl"] Nov 03 17:09:18.692370 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.692211 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-8558rpcjl"] Nov 03 17:09:18.859188 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.859159 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" Nov 03 17:09:18.943750 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.943714 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f011e68-26d2-40c9-b439-550813b3fdc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f011e68-26d2-40c9-b439-550813b3fdc9-plugins-config-volume\") pod \"2f011e68-26d2-40c9-b439-550813b3fdc9\" (UID: \"2f011e68-26d2-40c9-b439-550813b3fdc9\") " Nov 03 17:09:18.943934 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.943855 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f011e68-26d2-40c9-b439-550813b3fdc9-kube-api-access-rclnj\" (UniqueName: \"kubernetes.io/projected/2f011e68-26d2-40c9-b439-550813b3fdc9-kube-api-access-rclnj\") pod \"2f011e68-26d2-40c9-b439-550813b3fdc9\" (UID: \"2f011e68-26d2-40c9-b439-550813b3fdc9\") " Nov 03 17:09:18.944009 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.943953 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f011e68-26d2-40c9-b439-550813b3fdc9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f011e68-26d2-40c9-b439-550813b3fdc9-provider-credentials\") pod \"2f011e68-26d2-40c9-b439-550813b3fdc9\" (UID: \"2f011e68-26d2-40c9-b439-550813b3fdc9\") " Nov 03 17:09:18.944009 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.943990 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f011e68-26d2-40c9-b439-550813b3fdc9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f011e68-26d2-40c9-b439-550813b3fdc9-routing-overlay-volume\") pod \"2f011e68-26d2-40c9-b439-550813b3fdc9\" (UID: \"2f011e68-26d2-40c9-b439-550813b3fdc9\") " Nov 03 17:09:18.944164 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.944127 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f011e68-26d2-40c9-b439-550813b3fdc9-plugins-config-volume" pod "2f011e68-26d2-40c9-b439-550813b3fdc9" (UID: "2f011e68-26d2-40c9-b439-550813b3fdc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:18.944628 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.944598 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f011e68-26d2-40c9-b439-550813b3fdc9-routing-overlay-volume" pod "2f011e68-26d2-40c9-b439-550813b3fdc9" (UID: "2f011e68-26d2-40c9-b439-550813b3fdc9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:18.944887 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.944866 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f011e68-26d2-40c9-b439-550813b3fdc9-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:18.944887 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.944888 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f011e68-26d2-40c9-b439-550813b3fdc9-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:18.947111 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.947088 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f011e68-26d2-40c9-b439-550813b3fdc9-kube-api-access-rclnj" pod "2f011e68-26d2-40c9-b439-550813b3fdc9" (UID: "2f011e68-26d2-40c9-b439-550813b3fdc9"). InnerVolumeSpecName "kube-api-access-rclnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:18.947243 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:18.947124 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f011e68-26d2-40c9-b439-550813b3fdc9-provider-credentials" pod "2f011e68-26d2-40c9-b439-550813b3fdc9" (UID: "2f011e68-26d2-40c9-b439-550813b3fdc9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:19.046340 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:19.046307 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f011e68-26d2-40c9-b439-550813b3fdc9-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:19.046340 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:19.046337 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rclnj\" (UniqueName: \"kubernetes.io/projected/2f011e68-26d2-40c9-b439-550813b3fdc9-kube-api-access-rclnj\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:19.632063 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:19.632029 4947 generic.go:356] "Generic (PLEG): container finished" podID="2f011e68-26d2-40c9-b439-550813b3fdc9" containerID="858b7a5f5da2ef0c107e6228973bc37bed51c2c13f0f72b9202d2fda1b1d2350" exitCode=0 Nov 03 17:09:19.632551 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:19.632074 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" Nov 03 17:09:19.632551 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:19.632104 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" event={"ID":"2f011e68-26d2-40c9-b439-550813b3fdc9","Type":"ContainerDied","Data":"858b7a5f5da2ef0c107e6228973bc37bed51c2c13f0f72b9202d2fda1b1d2350"} Nov 03 17:09:19.632551 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:19.632163 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j" event={"ID":"2f011e68-26d2-40c9-b439-550813b3fdc9","Type":"ContainerDied","Data":"6536b952cee49daa0d87bcc10f67acd52db243f09d0ab5e3fa373fccc18f012b"} Nov 03 17:09:19.632551 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:19.632194 4947 scope.go:122] "RemoveContainer" containerID="858b7a5f5da2ef0c107e6228973bc37bed51c2c13f0f72b9202d2fda1b1d2350" Nov 03 17:09:19.644735 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:19.644689 4947 scope.go:122] "RemoveContainer" containerID="858b7a5f5da2ef0c107e6228973bc37bed51c2c13f0f72b9202d2fda1b1d2350" Nov 03 17:09:19.645067 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:19.645046 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858b7a5f5da2ef0c107e6228973bc37bed51c2c13f0f72b9202d2fda1b1d2350\": container with ID starting with 858b7a5f5da2ef0c107e6228973bc37bed51c2c13f0f72b9202d2fda1b1d2350 not found: ID does not exist" containerID="858b7a5f5da2ef0c107e6228973bc37bed51c2c13f0f72b9202d2fda1b1d2350" Nov 03 17:09:19.645150 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:19.645076 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858b7a5f5da2ef0c107e6228973bc37bed51c2c13f0f72b9202d2fda1b1d2350"} err="failed to get container status \"858b7a5f5da2ef0c107e6228973bc37bed51c2c13f0f72b9202d2fda1b1d2350\": rpc error: code = NotFound desc = could not find container \"858b7a5f5da2ef0c107e6228973bc37bed51c2c13f0f72b9202d2fda1b1d2350\": container with ID starting with 858b7a5f5da2ef0c107e6228973bc37bed51c2c13f0f72b9202d2fda1b1d2350 not found: ID does not exist" Nov 03 17:09:19.689981 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:19.689944 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j"] Nov 03 17:09:19.695048 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:19.695023 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75g2m6j"] Nov 03 17:09:20.283444 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:20.283409 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-2492c4/agc-cred-4fb82f7c0e56: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:09:20.283444 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:20.283436 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x: secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:09:20.283677 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:20.283495 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-provider-credentials podName:df509afa-08d4-485f-8fbc-dc80c0bbec6e nodeName:}" failed. No retries permitted until 2026-11-03 17:09:24.283480456 +0000 UTC m=+3307.872480415 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" (UID: "df509afa-08d4-485f-8fbc-dc80c0bbec6e") : secret "agc-cred-4fb82f7c0e56" not found Nov 03 17:09:20.385270 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:20.385230 4947 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-plugins-e2e-extmod-a-2492c4: configmap "payload-processing-external-model-plugins-e2e-extmod-a-2492c4" not found Nov 03 17:09:20.385472 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:20.385306 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-plugins-config-volume podName:df509afa-08d4-485f-8fbc-dc80c0bbec6e nodeName:}" failed. No retries permitted until 2026-11-03 17:09:24.385290774 +0000 UTC m=+3307.974290738 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" (UID: "df509afa-08d4-485f-8fbc-dc80c0bbec6e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-2492c4" not found Nov 03 17:09:20.578916 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:20.578822 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f011e68-26d2-40c9-b439-550813b3fdc9" path="/var/lib/kubelet/pods/2f011e68-26d2-40c9-b439-550813b3fdc9/volumes" Nov 03 17:09:20.579327 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:20.579312 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4cda53a-2726-4dc0-8057-f3302901c980" path="/var/lib/kubelet/pods/d4cda53a-2726-4dc0-8057-f3302901c980/volumes" Nov 03 17:09:22.622962 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:22.622913 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" podUID="df509afa-08d4-485f-8fbc-dc80c0bbec6e" containerName="payload-processing" containerID="cri-o://5894154601e062072aa5b0bbb052afebe47211f3a9639a848c20237b2dc6af31" gracePeriod=25 Nov 03 17:09:22.957839 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:22.957809 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" Nov 03 17:09:23.039961 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.039928 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-plugins-config-volume\") pod \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\" (UID: \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\") " Nov 03 17:09:23.040160 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.040044 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-routing-overlay-volume\") pod \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\" (UID: \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\") " Nov 03 17:09:23.040160 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.040120 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-provider-credentials\") pod \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\" (UID: \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\") " Nov 03 17:09:23.040160 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.040151 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-kube-api-access-zr6b4\" (UniqueName: \"kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-kube-api-access-zr6b4\") pod \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\" (UID: \"df509afa-08d4-485f-8fbc-dc80c0bbec6e\") " Nov 03 17:09:23.040428 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.040398 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-routing-overlay-volume" pod "df509afa-08d4-485f-8fbc-dc80c0bbec6e" (UID: "df509afa-08d4-485f-8fbc-dc80c0bbec6e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:23.040502 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.040440 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-plugins-config-volume" pod "df509afa-08d4-485f-8fbc-dc80c0bbec6e" (UID: "df509afa-08d4-485f-8fbc-dc80c0bbec6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:23.041157 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.041134 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:23.041267 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.041161 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df509afa-08d4-485f-8fbc-dc80c0bbec6e-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:23.042568 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.042543 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-provider-credentials" pod "df509afa-08d4-485f-8fbc-dc80c0bbec6e" (UID: "df509afa-08d4-485f-8fbc-dc80c0bbec6e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:23.042568 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.042577 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-kube-api-access-zr6b4" pod "df509afa-08d4-485f-8fbc-dc80c0bbec6e" (UID: "df509afa-08d4-485f-8fbc-dc80c0bbec6e"). InnerVolumeSpecName "kube-api-access-zr6b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:23.142953 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.142914 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:23.142953 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.142946 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr6b4\" (UniqueName: \"kubernetes.io/projected/df509afa-08d4-485f-8fbc-dc80c0bbec6e-kube-api-access-zr6b4\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:23.666169 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.666138 4947 generic.go:356] "Generic (PLEG): container finished" podID="df509afa-08d4-485f-8fbc-dc80c0bbec6e" containerID="5894154601e062072aa5b0bbb052afebe47211f3a9639a848c20237b2dc6af31" exitCode=0 Nov 03 17:09:23.666655 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.666182 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" Nov 03 17:09:23.666655 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.666219 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" event={"ID":"df509afa-08d4-485f-8fbc-dc80c0bbec6e","Type":"ContainerDied","Data":"5894154601e062072aa5b0bbb052afebe47211f3a9639a848c20237b2dc6af31"} Nov 03 17:09:23.666655 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.666273 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x" event={"ID":"df509afa-08d4-485f-8fbc-dc80c0bbec6e","Type":"ContainerDied","Data":"b1053743cdd28679db8d78170204b159fd86fd620afff1ef3f408f8436cdbf80"} Nov 03 17:09:23.666655 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.666296 4947 scope.go:122] "RemoveContainer" containerID="5894154601e062072aa5b0bbb052afebe47211f3a9639a848c20237b2dc6af31" Nov 03 17:09:23.677912 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.677883 4947 scope.go:122] "RemoveContainer" containerID="5894154601e062072aa5b0bbb052afebe47211f3a9639a848c20237b2dc6af31" Nov 03 17:09:23.678216 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:23.678197 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5894154601e062072aa5b0bbb052afebe47211f3a9639a848c20237b2dc6af31\": container with ID starting with 5894154601e062072aa5b0bbb052afebe47211f3a9639a848c20237b2dc6af31 not found: ID does not exist" containerID="5894154601e062072aa5b0bbb052afebe47211f3a9639a848c20237b2dc6af31" Nov 03 17:09:23.678262 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.678226 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5894154601e062072aa5b0bbb052afebe47211f3a9639a848c20237b2dc6af31"} err="failed to get container status \"5894154601e062072aa5b0bbb052afebe47211f3a9639a848c20237b2dc6af31\": rpc error: code = NotFound desc = could not find container \"5894154601e062072aa5b0bbb052afebe47211f3a9639a848c20237b2dc6af31\": container with ID starting with 5894154601e062072aa5b0bbb052afebe47211f3a9639a848c20237b2dc6af31 not found: ID does not exist" Nov 03 17:09:23.722945 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.722907 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x"] Nov 03 17:09:23.736961 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:23.736928 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-76b5mct5x"] Nov 03 17:09:24.567617 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:24.567584 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df509afa-08d4-485f-8fbc-dc80c0bbec6e" path="/var/lib/kubelet/pods/df509afa-08d4-485f-8fbc-dc80c0bbec6e/volumes" Nov 03 17:09:25.738521 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.738486 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78"] Nov 03 17:09:25.739291 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.739276 4947 cpu_manager.go:414] "containerMap: removing container" podUID="d4cda53a-2726-4dc0-8057-f3302901c980" containerName="payload-processing" Nov 03 17:09:25.739342 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.739293 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4cda53a-2726-4dc0-8057-f3302901c980" containerName="payload-processing" Nov 03 17:09:25.739342 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.739305 4947 cpu_manager.go:414] "containerMap: removing container" podUID="2f011e68-26d2-40c9-b439-550813b3fdc9" containerName="payload-processing" Nov 03 17:09:25.739342 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.739309 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f011e68-26d2-40c9-b439-550813b3fdc9" containerName="payload-processing" Nov 03 17:09:25.739342 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.739330 4947 cpu_manager.go:414] "containerMap: removing container" podUID="df509afa-08d4-485f-8fbc-dc80c0bbec6e" containerName="payload-processing" Nov 03 17:09:25.739342 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.739335 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df509afa-08d4-485f-8fbc-dc80c0bbec6e" containerName="payload-processing" Nov 03 17:09:25.739498 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.739435 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4cda53a-2726-4dc0-8057-f3302901c980" containerName="payload-processing" Nov 03 17:09:25.739498 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.739447 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f011e68-26d2-40c9-b439-550813b3fdc9" containerName="payload-processing" Nov 03 17:09:25.739498 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.739456 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="df509afa-08d4-485f-8fbc-dc80c0bbec6e" containerName="payload-processing" Nov 03 17:09:25.744106 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.744090 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" Nov 03 17:09:25.783289 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.783251 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78"] Nov 03 17:09:25.789876 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.789841 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78\" (UID: \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" Nov 03 17:09:25.790109 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.789908 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78\" (UID: \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" Nov 03 17:09:25.790246 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.790224 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78\" (UID: \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" Nov 03 17:09:25.790336 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.790304 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7d7\" (UniqueName: \"kubernetes.io/projected/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-kube-api-access-st7d7\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78\" (UID: \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" Nov 03 17:09:25.906729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.899154 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78\" (UID: \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" Nov 03 17:09:25.916722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.915428 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78\" (UID: \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" Nov 03 17:09:25.916722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.915540 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7d7\" (UniqueName: \"kubernetes.io/projected/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-kube-api-access-st7d7\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78\" (UID: \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" Nov 03 17:09:25.916722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.915630 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78\" (UID: \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" Nov 03 17:09:25.916722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.916654 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78\" (UID: \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" Nov 03 17:09:25.941723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.935170 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78\" (UID: \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" Nov 03 17:09:25.997777 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:25.967744 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78\" (UID: \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" Nov 03 17:09:26.074093 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:26.047793 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7d7\" (UniqueName: \"kubernetes.io/projected/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-kube-api-access-st7d7\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78\" (UID: \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" Nov 03 17:09:26.074093 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:26.065288 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" Nov 03 17:09:26.429391 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:09:26.429353 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e97fd3_c7f4_4ad9_bc44_08db6dce65a5.slice/crio-8e098f11e3a27fb654011498ba85f70482442c562fa37b1c900e7530c459bef0 WatchSource:0}: Error finding container 8e098f11e3a27fb654011498ba85f70482442c562fa37b1c900e7530c459bef0: Status 404 returned error can't find the container with id 8e098f11e3a27fb654011498ba85f70482442c562fa37b1c900e7530c459bef0 Nov 03 17:09:26.430169 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:26.430125 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78"] Nov 03 17:09:26.692888 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:26.692800 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" event={"ID":"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5","Type":"ContainerStarted","Data":"769f9af2bf55dea5dd3ab53923f0abd10f76904638ce644622b4353903c2424e"} Nov 03 17:09:26.692888 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:26.692851 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" event={"ID":"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5","Type":"ContainerStarted","Data":"8e098f11e3a27fb654011498ba85f70482442c562fa37b1c900e7530c459bef0"} Nov 03 17:09:26.693148 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:26.692919 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" Nov 03 17:09:26.741899 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:26.741844 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" podStartSLOduration=1.741830662 podStartE2EDuration="1.741830662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:09:26.738267847 +0000 UTC m=+3310.327267817" watchObservedRunningTime="2026-11-03 17:09:26.741830662 +0000 UTC m=+3310.330830630" Nov 03 17:09:27.228244 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:27.228198 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw"] Nov 03 17:09:27.231903 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:27.231875 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" Nov 03 17:09:27.241295 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:27.241263 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl"] Nov 03 17:09:30.141854 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:30.141817 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:09:32.030312 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.030266 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md"] Nov 03 17:09:32.038055 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.038024 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" Nov 03 17:09:32.056001 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.055975 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-ccc9e2c189bf\"" Nov 03 17:09:32.060112 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.060085 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md"] Nov 03 17:09:32.086780 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.069907 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 17:09:32.086780 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.070048 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-bxthl\"" Nov 03 17:09:32.086780 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.070161 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 17:09:32.086780 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.070215 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 17:09:32.137549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.135593 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpp6c\" (UniqueName: \"kubernetes.io/projected/df29aa6b-58f4-48b9-87b6-09b6b50260e6-kube-api-access-mpp6c\") pod \"payload-processing-external-model-6b65945b4c-6c6md\" (UID: \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\") " pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" Nov 03 17:09:32.147724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.144354 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df29aa6b-58f4-48b9-87b6-09b6b50260e6-routing-overlay-volume\") pod \"payload-processing-external-model-6b65945b4c-6c6md\" (UID: \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\") " pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" Nov 03 17:09:32.147724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.144504 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df29aa6b-58f4-48b9-87b6-09b6b50260e6-plugins-config-volume\") pod \"payload-processing-external-model-6b65945b4c-6c6md\" (UID: \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\") " pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" Nov 03 17:09:32.147724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.144587 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df29aa6b-58f4-48b9-87b6-09b6b50260e6-provider-credentials\") pod \"payload-processing-external-model-6b65945b4c-6c6md\" (UID: \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\") " pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" Nov 03 17:09:32.156996 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.156966 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 17:09:32.251726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.240786 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" podUID="8360f74a-698b-4eab-880f-e399f4ee23d7" containerName="payload-processing" containerID="cri-o://2e7cc9d908b990a3c0909bf053a563b0318f727ee4b50c0187d5d4c70de5ebf1" gracePeriod=25 Nov 03 17:09:32.251726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.241626 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" podUID="101a5bb6-0ff8-4498-82d1-c75ddbac14a1" containerName="payload-processing" containerID="cri-o://c9d52a341a327253ebdae7156b10714d462515015e31114c88202f758fa12eba" gracePeriod=25 Nov 03 17:09:32.252460 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.251900 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpp6c\" (UniqueName: \"kubernetes.io/projected/df29aa6b-58f4-48b9-87b6-09b6b50260e6-kube-api-access-mpp6c\") pod \"payload-processing-external-model-6b65945b4c-6c6md\" (UID: \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\") " pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" Nov 03 17:09:32.264721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.262124 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df29aa6b-58f4-48b9-87b6-09b6b50260e6-routing-overlay-volume\") pod \"payload-processing-external-model-6b65945b4c-6c6md\" (UID: \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\") " pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" Nov 03 17:09:32.264721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.262358 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df29aa6b-58f4-48b9-87b6-09b6b50260e6-plugins-config-volume\") pod \"payload-processing-external-model-6b65945b4c-6c6md\" (UID: \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\") " pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" Nov 03 17:09:32.264721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.262533 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df29aa6b-58f4-48b9-87b6-09b6b50260e6-provider-credentials\") pod \"payload-processing-external-model-6b65945b4c-6c6md\" (UID: \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\") " pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" Nov 03 17:09:32.307732 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.289612 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df29aa6b-58f4-48b9-87b6-09b6b50260e6-routing-overlay-volume\") pod \"payload-processing-external-model-6b65945b4c-6c6md\" (UID: \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\") " pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" Nov 03 17:09:32.307732 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.290893 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df29aa6b-58f4-48b9-87b6-09b6b50260e6-plugins-config-volume\") pod \"payload-processing-external-model-6b65945b4c-6c6md\" (UID: \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\") " pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" Nov 03 17:09:32.345764 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.308303 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df29aa6b-58f4-48b9-87b6-09b6b50260e6-provider-credentials\") pod \"payload-processing-external-model-6b65945b4c-6c6md\" (UID: \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\") " pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" Nov 03 17:09:32.361717 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.359945 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpp6c\" (UniqueName: \"kubernetes.io/projected/df29aa6b-58f4-48b9-87b6-09b6b50260e6-kube-api-access-mpp6c\") pod \"payload-processing-external-model-6b65945b4c-6c6md\" (UID: \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\") " pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" Nov 03 17:09:32.387721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.387208 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" Nov 03 17:09:32.667957 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.667928 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" Nov 03 17:09:32.678132 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.678102 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" Nov 03 17:09:32.694203 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.694173 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-plugins-config-volume\") pod \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\" (UID: \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\") " Nov 03 17:09:32.694369 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.694245 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8360f74a-698b-4eab-880f-e399f4ee23d7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8360f74a-698b-4eab-880f-e399f4ee23d7-routing-overlay-volume\") pod \"8360f74a-698b-4eab-880f-e399f4ee23d7\" (UID: \"8360f74a-698b-4eab-880f-e399f4ee23d7\") " Nov 03 17:09:32.694369 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.694305 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8360f74a-698b-4eab-880f-e399f4ee23d7-kube-api-access-4ph4c\" (UniqueName: \"kubernetes.io/projected/8360f74a-698b-4eab-880f-e399f4ee23d7-kube-api-access-4ph4c\") pod \"8360f74a-698b-4eab-880f-e399f4ee23d7\" (UID: \"8360f74a-698b-4eab-880f-e399f4ee23d7\") " Nov 03 17:09:32.694369 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.694326 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-provider-credentials\") pod \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\" (UID: \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\") " Nov 03 17:09:32.694631 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.694476 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-kube-api-access-h7rpd\" (UniqueName: \"kubernetes.io/projected/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-kube-api-access-h7rpd\") pod \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\" (UID: \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\") " Nov 03 17:09:32.694631 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.694600 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8360f74a-698b-4eab-880f-e399f4ee23d7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8360f74a-698b-4eab-880f-e399f4ee23d7-provider-credentials\") pod \"8360f74a-698b-4eab-880f-e399f4ee23d7\" (UID: \"8360f74a-698b-4eab-880f-e399f4ee23d7\") " Nov 03 17:09:32.694631 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.694602 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-plugins-config-volume" pod "101a5bb6-0ff8-4498-82d1-c75ddbac14a1" (UID: "101a5bb6-0ff8-4498-82d1-c75ddbac14a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:32.694878 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.694659 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8360f74a-698b-4eab-880f-e399f4ee23d7-routing-overlay-volume" pod "8360f74a-698b-4eab-880f-e399f4ee23d7" (UID: "8360f74a-698b-4eab-880f-e399f4ee23d7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:32.694878 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.694688 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-routing-overlay-volume\") pod \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\" (UID: \"101a5bb6-0ff8-4498-82d1-c75ddbac14a1\") " Nov 03 17:09:32.694878 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.694773 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8360f74a-698b-4eab-880f-e399f4ee23d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8360f74a-698b-4eab-880f-e399f4ee23d7-plugins-config-volume\") pod \"8360f74a-698b-4eab-880f-e399f4ee23d7\" (UID: \"8360f74a-698b-4eab-880f-e399f4ee23d7\") " Nov 03 17:09:32.695129 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.695100 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-routing-overlay-volume" pod "101a5bb6-0ff8-4498-82d1-c75ddbac14a1" (UID: "101a5bb6-0ff8-4498-82d1-c75ddbac14a1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:32.695248 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.695205 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8360f74a-698b-4eab-880f-e399f4ee23d7-plugins-config-volume" pod "8360f74a-698b-4eab-880f-e399f4ee23d7" (UID: "8360f74a-698b-4eab-880f-e399f4ee23d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:32.695517 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.695487 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:32.695517 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.695514 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8360f74a-698b-4eab-880f-e399f4ee23d7-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:32.695798 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.695528 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:32.695798 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.695541 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8360f74a-698b-4eab-880f-e399f4ee23d7-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:32.697039 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.697018 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-kube-api-access-h7rpd" pod "101a5bb6-0ff8-4498-82d1-c75ddbac14a1" (UID: "101a5bb6-0ff8-4498-82d1-c75ddbac14a1"). InnerVolumeSpecName "kube-api-access-h7rpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:32.697114 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.697079 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8360f74a-698b-4eab-880f-e399f4ee23d7-provider-credentials" pod "8360f74a-698b-4eab-880f-e399f4ee23d7" (UID: "8360f74a-698b-4eab-880f-e399f4ee23d7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:32.697274 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.697255 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8360f74a-698b-4eab-880f-e399f4ee23d7-kube-api-access-4ph4c" pod "8360f74a-698b-4eab-880f-e399f4ee23d7" (UID: "8360f74a-698b-4eab-880f-e399f4ee23d7"). InnerVolumeSpecName "kube-api-access-4ph4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:32.697338 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.697274 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-provider-credentials" pod "101a5bb6-0ff8-4498-82d1-c75ddbac14a1" (UID: "101a5bb6-0ff8-4498-82d1-c75ddbac14a1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:32.764760 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:09:32.764720 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf29aa6b_58f4_48b9_87b6_09b6b50260e6.slice/crio-855b1b0bdb5f77a1d43c5763dc03a93450f92b684ca87705b597bba7c4720427 WatchSource:0}: Error finding container 855b1b0bdb5f77a1d43c5763dc03a93450f92b684ca87705b597bba7c4720427: Status 404 returned error can't find the container with id 855b1b0bdb5f77a1d43c5763dc03a93450f92b684ca87705b597bba7c4720427 Nov 03 17:09:32.765974 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.765947 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md"] Nov 03 17:09:32.794630 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.794585 4947 generic.go:356] "Generic (PLEG): container finished" podID="8360f74a-698b-4eab-880f-e399f4ee23d7" containerID="2e7cc9d908b990a3c0909bf053a563b0318f727ee4b50c0187d5d4c70de5ebf1" exitCode=0 Nov 03 17:09:32.794904 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.794872 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" Nov 03 17:09:32.795290 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.794859 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" event={"ID":"8360f74a-698b-4eab-880f-e399f4ee23d7","Type":"ContainerDied","Data":"2e7cc9d908b990a3c0909bf053a563b0318f727ee4b50c0187d5d4c70de5ebf1"} Nov 03 17:09:32.795290 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.795117 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw" event={"ID":"8360f74a-698b-4eab-880f-e399f4ee23d7","Type":"ContainerDied","Data":"04a215f6a5a998fc57f545577fed983ca745b9aa25a5b57c815a8f178c078f71"} Nov 03 17:09:32.795290 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.795153 4947 scope.go:122] "RemoveContainer" containerID="2e7cc9d908b990a3c0909bf053a563b0318f727ee4b50c0187d5d4c70de5ebf1" Nov 03 17:09:32.798679 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.798650 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8360f74a-698b-4eab-880f-e399f4ee23d7-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:32.798847 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.798685 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ph4c\" (UniqueName: \"kubernetes.io/projected/8360f74a-698b-4eab-880f-e399f4ee23d7-kube-api-access-4ph4c\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:32.798847 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.798802 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:32.798847 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.798819 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7rpd\" (UniqueName: \"kubernetes.io/projected/101a5bb6-0ff8-4498-82d1-c75ddbac14a1-kube-api-access-h7rpd\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:32.799937 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.799862 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" event={"ID":"df29aa6b-58f4-48b9-87b6-09b6b50260e6","Type":"ContainerStarted","Data":"855b1b0bdb5f77a1d43c5763dc03a93450f92b684ca87705b597bba7c4720427"} Nov 03 17:09:32.802750 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.801601 4947 generic.go:356] "Generic (PLEG): container finished" podID="101a5bb6-0ff8-4498-82d1-c75ddbac14a1" containerID="c9d52a341a327253ebdae7156b10714d462515015e31114c88202f758fa12eba" exitCode=0 Nov 03 17:09:32.802750 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.801673 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" Nov 03 17:09:32.802750 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.801692 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" event={"ID":"101a5bb6-0ff8-4498-82d1-c75ddbac14a1","Type":"ContainerDied","Data":"c9d52a341a327253ebdae7156b10714d462515015e31114c88202f758fa12eba"} Nov 03 17:09:32.802750 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.801789 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl" event={"ID":"101a5bb6-0ff8-4498-82d1-c75ddbac14a1","Type":"ContainerDied","Data":"b25cdb46c422f192bd48a932c0491b94602910331b98a78301c2c2ee7e23cab2"} Nov 03 17:09:32.816145 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.816122 4947 scope.go:122] "RemoveContainer" containerID="2e7cc9d908b990a3c0909bf053a563b0318f727ee4b50c0187d5d4c70de5ebf1" Nov 03 17:09:32.816547 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:32.816519 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7cc9d908b990a3c0909bf053a563b0318f727ee4b50c0187d5d4c70de5ebf1\": container with ID starting with 2e7cc9d908b990a3c0909bf053a563b0318f727ee4b50c0187d5d4c70de5ebf1 not found: ID does not exist" containerID="2e7cc9d908b990a3c0909bf053a563b0318f727ee4b50c0187d5d4c70de5ebf1" Nov 03 17:09:32.816669 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.816562 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7cc9d908b990a3c0909bf053a563b0318f727ee4b50c0187d5d4c70de5ebf1"} err="failed to get container status \"2e7cc9d908b990a3c0909bf053a563b0318f727ee4b50c0187d5d4c70de5ebf1\": rpc error: code = NotFound desc = could not find container \"2e7cc9d908b990a3c0909bf053a563b0318f727ee4b50c0187d5d4c70de5ebf1\": container with ID starting with 2e7cc9d908b990a3c0909bf053a563b0318f727ee4b50c0187d5d4c70de5ebf1 not found: ID does not exist" Nov 03 17:09:32.816669 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.816589 4947 scope.go:122] "RemoveContainer" containerID="c9d52a341a327253ebdae7156b10714d462515015e31114c88202f758fa12eba" Nov 03 17:09:32.836660 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.836630 4947 scope.go:122] "RemoveContainer" containerID="c9d52a341a327253ebdae7156b10714d462515015e31114c88202f758fa12eba" Nov 03 17:09:32.837068 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:32.837048 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d52a341a327253ebdae7156b10714d462515015e31114c88202f758fa12eba\": container with ID starting with c9d52a341a327253ebdae7156b10714d462515015e31114c88202f758fa12eba not found: ID does not exist" containerID="c9d52a341a327253ebdae7156b10714d462515015e31114c88202f758fa12eba" Nov 03 17:09:32.837169 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.837079 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d52a341a327253ebdae7156b10714d462515015e31114c88202f758fa12eba"} err="failed to get container status \"c9d52a341a327253ebdae7156b10714d462515015e31114c88202f758fa12eba\": rpc error: code = NotFound desc = could not find container \"c9d52a341a327253ebdae7156b10714d462515015e31114c88202f758fa12eba\": container with ID starting with c9d52a341a327253ebdae7156b10714d462515015e31114c88202f758fa12eba not found: ID does not exist" Nov 03 17:09:32.873906 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.873873 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw"] Nov 03 17:09:32.878232 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.878199 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-cb8clx2tw"] Nov 03 17:09:32.929010 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.928969 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl"] Nov 03 17:09:32.934247 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:32.934216 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ca9d7c/payload-processing-external-model-e2e-extmod-b-ca9d7c-55c44qvdl"] Nov 03 17:09:33.812140 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:33.812103 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" event={"ID":"df29aa6b-58f4-48b9-87b6-09b6b50260e6","Type":"ContainerStarted","Data":"faa84332b93fc6bc1cc5224940c853c16b38e0aec940a48a443b8559bc928454"} Nov 03 17:09:33.812670 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:33.812344 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" Nov 03 17:09:33.857869 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:33.857813 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" podStartSLOduration=1.857794597 podStartE2EDuration="1.857794597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:09:33.857195862 +0000 UTC m=+3317.446195832" watchObservedRunningTime="2026-11-03 17:09:33.857794597 +0000 UTC m=+3317.446794754" Nov 03 17:09:34.566898 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:34.566867 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101a5bb6-0ff8-4498-82d1-c75ddbac14a1" path="/var/lib/kubelet/pods/101a5bb6-0ff8-4498-82d1-c75ddbac14a1/volumes" Nov 03 17:09:34.567332 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:34.567318 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8360f74a-698b-4eab-880f-e399f4ee23d7" path="/var/lib/kubelet/pods/8360f74a-698b-4eab-880f-e399f4ee23d7/volumes" Nov 03 17:09:37.628826 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.628784 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78"] Nov 03 17:09:37.632147 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.632121 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" Nov 03 17:09:37.706121 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.706082 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx"] Nov 03 17:09:37.707042 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.707014 4947 cpu_manager.go:414] "containerMap: removing container" podUID="101a5bb6-0ff8-4498-82d1-c75ddbac14a1" containerName="payload-processing" Nov 03 17:09:37.707042 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.707041 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101a5bb6-0ff8-4498-82d1-c75ddbac14a1" containerName="payload-processing" Nov 03 17:09:37.707242 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.707063 4947 cpu_manager.go:414] "containerMap: removing container" podUID="8360f74a-698b-4eab-880f-e399f4ee23d7" containerName="payload-processing" Nov 03 17:09:37.707242 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.707070 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8360f74a-698b-4eab-880f-e399f4ee23d7" containerName="payload-processing" Nov 03 17:09:37.707349 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.707319 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="101a5bb6-0ff8-4498-82d1-c75ddbac14a1" containerName="payload-processing" Nov 03 17:09:37.707349 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.707337 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="8360f74a-698b-4eab-880f-e399f4ee23d7" containerName="payload-processing" Nov 03 17:09:37.712765 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.712732 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" Nov 03 17:09:37.742187 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.742138 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx"] Nov 03 17:09:37.785318 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.785286 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e61163db-7a96-4fba-b15c-956a8cabe4de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx\" (UID: \"e61163db-7a96-4fba-b15c-956a8cabe4de\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" Nov 03 17:09:37.785318 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.785324 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e61163db-7a96-4fba-b15c-956a8cabe4de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx\" (UID: \"e61163db-7a96-4fba-b15c-956a8cabe4de\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" Nov 03 17:09:37.785566 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.785362 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xff27\" (UniqueName: \"kubernetes.io/projected/e61163db-7a96-4fba-b15c-956a8cabe4de-kube-api-access-xff27\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx\" (UID: \"e61163db-7a96-4fba-b15c-956a8cabe4de\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" Nov 03 17:09:37.785617 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.785574 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e61163db-7a96-4fba-b15c-956a8cabe4de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx\" (UID: \"e61163db-7a96-4fba-b15c-956a8cabe4de\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" Nov 03 17:09:37.896790 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.896488 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e61163db-7a96-4fba-b15c-956a8cabe4de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx\" (UID: \"e61163db-7a96-4fba-b15c-956a8cabe4de\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" Nov 03 17:09:37.904722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.897046 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e61163db-7a96-4fba-b15c-956a8cabe4de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx\" (UID: \"e61163db-7a96-4fba-b15c-956a8cabe4de\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" Nov 03 17:09:37.904722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.897094 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e61163db-7a96-4fba-b15c-956a8cabe4de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx\" (UID: \"e61163db-7a96-4fba-b15c-956a8cabe4de\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" Nov 03 17:09:37.904722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.897141 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xff27\" (UniqueName: \"kubernetes.io/projected/e61163db-7a96-4fba-b15c-956a8cabe4de-kube-api-access-xff27\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx\" (UID: \"e61163db-7a96-4fba-b15c-956a8cabe4de\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" Nov 03 17:09:37.915723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.909042 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e61163db-7a96-4fba-b15c-956a8cabe4de-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx\" (UID: \"e61163db-7a96-4fba-b15c-956a8cabe4de\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" Nov 03 17:09:37.915723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.909916 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e61163db-7a96-4fba-b15c-956a8cabe4de-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx\" (UID: \"e61163db-7a96-4fba-b15c-956a8cabe4de\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" Nov 03 17:09:37.937724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.937121 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xff27\" (UniqueName: \"kubernetes.io/projected/e61163db-7a96-4fba-b15c-956a8cabe4de-kube-api-access-xff27\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx\" (UID: \"e61163db-7a96-4fba-b15c-956a8cabe4de\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" Nov 03 17:09:37.941720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:37.938619 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e61163db-7a96-4fba-b15c-956a8cabe4de-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx\" (UID: \"e61163db-7a96-4fba-b15c-956a8cabe4de\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" Nov 03 17:09:38.029061 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:38.029029 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" Nov 03 17:09:38.361806 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:09:38.361771 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode61163db_7a96_4fba_b15c_956a8cabe4de.slice/crio-9d7a3e98525248739f11c622252d41b1cfbf548e22b6f50c3dc1411541b8775b WatchSource:0}: Error finding container 9d7a3e98525248739f11c622252d41b1cfbf548e22b6f50c3dc1411541b8775b: Status 404 returned error can't find the container with id 9d7a3e98525248739f11c622252d41b1cfbf548e22b6f50c3dc1411541b8775b Nov 03 17:09:38.362466 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:38.362442 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx"] Nov 03 17:09:38.866128 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:38.866095 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" event={"ID":"e61163db-7a96-4fba-b15c-956a8cabe4de","Type":"ContainerStarted","Data":"ea37ed5c2a16268ddba14d712984493821b74b68b03733621664f572aa708134"} Nov 03 17:09:38.866533 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:38.866148 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" event={"ID":"e61163db-7a96-4fba-b15c-956a8cabe4de","Type":"ContainerStarted","Data":"9d7a3e98525248739f11c622252d41b1cfbf548e22b6f50c3dc1411541b8775b"} Nov 03 17:09:38.866533 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:38.866175 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" Nov 03 17:09:38.914818 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:38.914763 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" podStartSLOduration=1.914749915 podStartE2EDuration="1.914749915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:09:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:09:38.912315599 +0000 UTC m=+3322.501315569" watchObservedRunningTime="2026-11-03 17:09:38.914749915 +0000 UTC m=+3322.503749878" Nov 03 17:09:42.630108 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:42.630023 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" podUID="04e97fd3-c7f4-4ad9-bc44-08db6dce65a5" containerName="payload-processing" containerID="cri-o://769f9af2bf55dea5dd3ab53923f0abd10f76904638ce644622b4353903c2424e" gracePeriod=25 Nov 03 17:09:42.914740 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:42.913270 4947 generic.go:356] "Generic (PLEG): container finished" podID="04e97fd3-c7f4-4ad9-bc44-08db6dce65a5" containerID="769f9af2bf55dea5dd3ab53923f0abd10f76904638ce644622b4353903c2424e" exitCode=0 Nov 03 17:09:42.914740 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:42.913356 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" event={"ID":"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5","Type":"ContainerDied","Data":"769f9af2bf55dea5dd3ab53923f0abd10f76904638ce644622b4353903c2424e"} Nov 03 17:09:43.006872 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:43.006842 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" Nov 03 17:09:43.105169 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:43.105133 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-kube-api-access-st7d7\" (UniqueName: \"kubernetes.io/projected/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-kube-api-access-st7d7\") pod \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\" (UID: \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\") " Nov 03 17:09:43.105375 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:43.105309 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-routing-overlay-volume\") pod \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\" (UID: \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\") " Nov 03 17:09:43.105544 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:43.105518 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-provider-credentials\") pod \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\" (UID: \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\") " Nov 03 17:09:43.105658 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:43.105630 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-plugins-config-volume\") pod \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\" (UID: \"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5\") " Nov 03 17:09:43.105799 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:43.105738 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-routing-overlay-volume" pod "04e97fd3-c7f4-4ad9-bc44-08db6dce65a5" (UID: "04e97fd3-c7f4-4ad9-bc44-08db6dce65a5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:43.106345 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:43.106318 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-plugins-config-volume" pod "04e97fd3-c7f4-4ad9-bc44-08db6dce65a5" (UID: "04e97fd3-c7f4-4ad9-bc44-08db6dce65a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:43.106739 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:43.106715 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:43.106861 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:43.106752 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:43.108025 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:43.107998 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-kube-api-access-st7d7" pod "04e97fd3-c7f4-4ad9-bc44-08db6dce65a5" (UID: "04e97fd3-c7f4-4ad9-bc44-08db6dce65a5"). InnerVolumeSpecName "kube-api-access-st7d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:43.108792 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:43.108759 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-provider-credentials" pod "04e97fd3-c7f4-4ad9-bc44-08db6dce65a5" (UID: "04e97fd3-c7f4-4ad9-bc44-08db6dce65a5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:43.208017 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:43.207932 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st7d7\" (UniqueName: \"kubernetes.io/projected/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-kube-api-access-st7d7\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:43.208017 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:43.207965 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:43.924602 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:43.924569 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" Nov 03 17:09:43.925134 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:43.924598 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78" event={"ID":"04e97fd3-c7f4-4ad9-bc44-08db6dce65a5","Type":"ContainerDied","Data":"8e098f11e3a27fb654011498ba85f70482442c562fa37b1c900e7530c459bef0"} Nov 03 17:09:43.925134 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:43.924676 4947 scope.go:122] "RemoveContainer" containerID="769f9af2bf55dea5dd3ab53923f0abd10f76904638ce644622b4353903c2424e" Nov 03 17:09:44.170937 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.102755 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt"] Nov 03 17:09:44.170937 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.104112 4947 cpu_manager.go:414] "containerMap: removing container" podUID="04e97fd3-c7f4-4ad9-bc44-08db6dce65a5" containerName="payload-processing" Nov 03 17:09:44.170937 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.104136 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e97fd3-c7f4-4ad9-bc44-08db6dce65a5" containerName="payload-processing" Nov 03 17:09:44.170937 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.104328 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e97fd3-c7f4-4ad9-bc44-08db6dce65a5" containerName="payload-processing" Nov 03 17:09:44.425363 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.421496 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt"] Nov 03 17:09:44.425363 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.421557 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78"] Nov 03 17:09:44.425363 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.421588 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-78488xx78"] Nov 03 17:09:44.425363 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.421716 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" Nov 03 17:09:44.439562 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.439527 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"agc-cred-edddadb39aba\"" Nov 03 17:09:44.439769 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.439585 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"payload-processing-external-model-e2e--e145110a-dockercfg-lpr5l\"" Nov 03 17:09:44.439858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.439826 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"routing-overlay\"" Nov 03 17:09:44.439931 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.439863 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"kube-root-ca.crt\"" Nov 03 17:09:44.439992 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.439950 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"openshift-service-ca.crt\"" Nov 03 17:09:44.440175 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.440155 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d6fcae\"" Nov 03 17:09:44.495319 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.495287 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5daa154-497b-4bfe-98ce-1c0af39b0224-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt\" (UID: \"e5daa154-497b-4bfe-98ce-1c0af39b0224\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" Nov 03 17:09:44.495539 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.495481 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5daa154-497b-4bfe-98ce-1c0af39b0224-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt\" (UID: \"e5daa154-497b-4bfe-98ce-1c0af39b0224\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" Nov 03 17:09:44.495622 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.495541 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5daa154-497b-4bfe-98ce-1c0af39b0224-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt\" (UID: \"e5daa154-497b-4bfe-98ce-1c0af39b0224\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" Nov 03 17:09:44.495684 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.495655 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4b24\" (UniqueName: \"kubernetes.io/projected/e5daa154-497b-4bfe-98ce-1c0af39b0224-kube-api-access-c4b24\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt\" (UID: \"e5daa154-497b-4bfe-98ce-1c0af39b0224\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" Nov 03 17:09:44.567929 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.567891 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e97fd3-c7f4-4ad9-bc44-08db6dce65a5" path="/var/lib/kubelet/pods/04e97fd3-c7f4-4ad9-bc44-08db6dce65a5/volumes" Nov 03 17:09:44.596763 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.596689 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5daa154-497b-4bfe-98ce-1c0af39b0224-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt\" (UID: \"e5daa154-497b-4bfe-98ce-1c0af39b0224\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" Nov 03 17:09:44.596984 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.596903 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5daa154-497b-4bfe-98ce-1c0af39b0224-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt\" (UID: \"e5daa154-497b-4bfe-98ce-1c0af39b0224\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" Nov 03 17:09:44.596984 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.596949 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5daa154-497b-4bfe-98ce-1c0af39b0224-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt\" (UID: \"e5daa154-497b-4bfe-98ce-1c0af39b0224\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" Nov 03 17:09:44.597105 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.597052 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4b24\" (UniqueName: \"kubernetes.io/projected/e5daa154-497b-4bfe-98ce-1c0af39b0224-kube-api-access-c4b24\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt\" (UID: \"e5daa154-497b-4bfe-98ce-1c0af39b0224\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" Nov 03 17:09:44.597552 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.597501 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5daa154-497b-4bfe-98ce-1c0af39b0224-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt\" (UID: \"e5daa154-497b-4bfe-98ce-1c0af39b0224\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" Nov 03 17:09:44.597775 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.597758 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5daa154-497b-4bfe-98ce-1c0af39b0224-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt\" (UID: \"e5daa154-497b-4bfe-98ce-1c0af39b0224\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" Nov 03 17:09:44.599660 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.599641 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5daa154-497b-4bfe-98ce-1c0af39b0224-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt\" (UID: \"e5daa154-497b-4bfe-98ce-1c0af39b0224\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" Nov 03 17:09:44.622986 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.622951 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4b24\" (UniqueName: \"kubernetes.io/projected/e5daa154-497b-4bfe-98ce-1c0af39b0224-kube-api-access-c4b24\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt\" (UID: \"e5daa154-497b-4bfe-98ce-1c0af39b0224\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" Nov 03 17:09:44.752651 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.752614 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" Nov 03 17:09:44.827297 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:44.825865 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" Nov 03 17:09:45.070181 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:09:45.070150 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5daa154_497b_4bfe_98ce_1c0af39b0224.slice/crio-a6e12248d613b140ffcfe7933c5f11a041b37ac15881eed21111a11bd7e9fbae WatchSource:0}: Error finding container a6e12248d613b140ffcfe7933c5f11a041b37ac15881eed21111a11bd7e9fbae: Status 404 returned error can't find the container with id a6e12248d613b140ffcfe7933c5f11a041b37ac15881eed21111a11bd7e9fbae Nov 03 17:09:45.070605 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.070463 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt"] Nov 03 17:09:45.492187 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.492159 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d"] Nov 03 17:09:45.497025 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.496989 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" Nov 03 17:09:45.514190 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.514159 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d6fcae\"/\"agc-cred-32f3e3812c29\"" Nov 03 17:09:45.519453 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.519420 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d"] Nov 03 17:09:45.611459 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.611424 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97v9j\" (UniqueName: \"kubernetes.io/projected/c6b80942-2b7a-4d66-a700-bbd1c642257e-kube-api-access-97v9j\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d\" (UID: \"c6b80942-2b7a-4d66-a700-bbd1c642257e\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" Nov 03 17:09:45.611459 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.611464 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6b80942-2b7a-4d66-a700-bbd1c642257e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d\" (UID: \"c6b80942-2b7a-4d66-a700-bbd1c642257e\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" Nov 03 17:09:45.611677 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.611639 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6b80942-2b7a-4d66-a700-bbd1c642257e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d\" (UID: \"c6b80942-2b7a-4d66-a700-bbd1c642257e\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" Nov 03 17:09:45.611677 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.611668 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b80942-2b7a-4d66-a700-bbd1c642257e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d\" (UID: \"c6b80942-2b7a-4d66-a700-bbd1c642257e\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" Nov 03 17:09:45.712980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.712939 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97v9j\" (UniqueName: \"kubernetes.io/projected/c6b80942-2b7a-4d66-a700-bbd1c642257e-kube-api-access-97v9j\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d\" (UID: \"c6b80942-2b7a-4d66-a700-bbd1c642257e\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" Nov 03 17:09:45.713145 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.712985 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6b80942-2b7a-4d66-a700-bbd1c642257e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d\" (UID: \"c6b80942-2b7a-4d66-a700-bbd1c642257e\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" Nov 03 17:09:45.713145 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.713119 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6b80942-2b7a-4d66-a700-bbd1c642257e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d\" (UID: \"c6b80942-2b7a-4d66-a700-bbd1c642257e\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" Nov 03 17:09:45.713287 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.713159 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b80942-2b7a-4d66-a700-bbd1c642257e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d\" (UID: \"c6b80942-2b7a-4d66-a700-bbd1c642257e\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" Nov 03 17:09:45.713944 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.713919 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b80942-2b7a-4d66-a700-bbd1c642257e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d\" (UID: \"c6b80942-2b7a-4d66-a700-bbd1c642257e\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" Nov 03 17:09:45.714077 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.713922 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6b80942-2b7a-4d66-a700-bbd1c642257e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d\" (UID: \"c6b80942-2b7a-4d66-a700-bbd1c642257e\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" Nov 03 17:09:45.716037 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.716016 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6b80942-2b7a-4d66-a700-bbd1c642257e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d\" (UID: \"c6b80942-2b7a-4d66-a700-bbd1c642257e\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" Nov 03 17:09:45.739140 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.739107 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97v9j\" (UniqueName: \"kubernetes.io/projected/c6b80942-2b7a-4d66-a700-bbd1c642257e-kube-api-access-97v9j\") pod \"payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d\" (UID: \"c6b80942-2b7a-4d66-a700-bbd1c642257e\") " pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" Nov 03 17:09:45.812806 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.812725 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" Nov 03 17:09:45.953587 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.953548 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" event={"ID":"e5daa154-497b-4bfe-98ce-1c0af39b0224","Type":"ContainerStarted","Data":"76bef5801000024a1126b6d65fed2216b5c7c227dc615789690ff6c9423040b1"} Nov 03 17:09:45.953890 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.953618 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" event={"ID":"e5daa154-497b-4bfe-98ce-1c0af39b0224","Type":"ContainerStarted","Data":"a6e12248d613b140ffcfe7933c5f11a041b37ac15881eed21111a11bd7e9fbae"} Nov 03 17:09:45.954764 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:45.954738 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" Nov 03 17:09:46.004007 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:46.003943 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" podStartSLOduration=3.003922906 podStartE2EDuration="3.003922906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:09:45.998781984 +0000 UTC m=+3329.587781955" watchObservedRunningTime="2026-11-03 17:09:46.003922906 +0000 UTC m=+3329.592922876" Nov 03 17:09:46.159146 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:09:46.159093 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b80942_2b7a_4d66_a700_bbd1c642257e.slice/crio-2aa6dd62c7bfe76f0c672682f1d66e663dbe3970897272ceffd08f8b2aaaf8a5 WatchSource:0}: Error finding container 2aa6dd62c7bfe76f0c672682f1d66e663dbe3970897272ceffd08f8b2aaaf8a5: Status 404 returned error can't find the container with id 2aa6dd62c7bfe76f0c672682f1d66e663dbe3970897272ceffd08f8b2aaaf8a5 Nov 03 17:09:46.161839 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:46.161747 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d"] Nov 03 17:09:46.963870 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:46.963675 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" event={"ID":"c6b80942-2b7a-4d66-a700-bbd1c642257e","Type":"ContainerStarted","Data":"6dc38dea6b116263af1f54d5349a990f876154f950aa1ac56c7dc64b7f92e6b0"} Nov 03 17:09:46.963870 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:46.963761 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" Nov 03 17:09:46.963870 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:46.963787 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" event={"ID":"c6b80942-2b7a-4d66-a700-bbd1c642257e","Type":"ContainerStarted","Data":"2aa6dd62c7bfe76f0c672682f1d66e663dbe3970897272ceffd08f8b2aaaf8a5"} Nov 03 17:09:47.012782 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:47.012722 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" podStartSLOduration=2.012689901 podStartE2EDuration="2.012689901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:09:47.007951489 +0000 UTC m=+3330.596951462" watchObservedRunningTime="2026-11-03 17:09:47.012689901 +0000 UTC m=+3330.601689870" Nov 03 17:09:48.974247 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:48.974136 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh"] Nov 03 17:09:48.984641 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:48.984608 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx"] Nov 03 17:09:48.989259 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:48.989231 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" Nov 03 17:09:51.605372 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.605334 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6"] Nov 03 17:09:51.610309 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.610288 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.627920 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.627893 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-3a0e27-openshift-default-dockercfg-s7k4t\"" Nov 03 17:09:51.646218 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.646174 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6"] Nov 03 17:09:51.744063 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.744022 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-credential-socket\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.744260 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.744127 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-istio-envoy\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.744260 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.744196 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-workload-certs\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.744388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.744276 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c02310-bff9-4d86-b641-8022d891b395-istio-token\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.744447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.744411 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-workload-socket\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.744538 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.744521 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjrr9\" (UniqueName: \"kubernetes.io/projected/e7c02310-bff9-4d86-b641-8022d891b395-kube-api-access-cjrr9\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.744597 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.744567 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c02310-bff9-4d86-b641-8022d891b395-istiod-ca-cert\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.744689 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.744674 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-istio-data\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.744829 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.744813 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c02310-bff9-4d86-b641-8022d891b395-istio-podinfo\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.845930 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.845889 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-istio-data\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.846122 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.845967 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c02310-bff9-4d86-b641-8022d891b395-istio-podinfo\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.846122 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.846068 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-credential-socket\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.846122 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.846109 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-istio-envoy\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.846321 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.846135 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-workload-certs\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.846321 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.846222 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c02310-bff9-4d86-b641-8022d891b395-istio-token\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.846321 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.846299 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-workload-socket\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.846486 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.846379 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjrr9\" (UniqueName: \"kubernetes.io/projected/e7c02310-bff9-4d86-b641-8022d891b395-kube-api-access-cjrr9\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.846486 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.846396 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-istio-data\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.846486 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.846421 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c02310-bff9-4d86-b641-8022d891b395-istiod-ca-cert\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.846669 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.846634 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-workload-socket\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.846788 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.846751 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-workload-certs\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.847005 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.846984 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-credential-socket\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.847321 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.847298 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c02310-bff9-4d86-b641-8022d891b395-istiod-ca-cert\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.848823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.848787 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-istio-envoy\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.849037 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.849019 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c02310-bff9-4d86-b641-8022d891b395-istio-podinfo\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.871545 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.871468 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c02310-bff9-4d86-b641-8022d891b395-istio-token\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.871682 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.871599 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjrr9\" (UniqueName: \"kubernetes.io/projected/e7c02310-bff9-4d86-b641-8022d891b395-kube-api-access-cjrr9\") pod \"e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:51.925986 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:51.925944 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:52.259124 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:09:52.259083 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7c02310_bff9_4d86_b641_8022d891b395.slice/crio-449dedc0ec25986c870defdf07c4124f76d72711164ed82b3baaabed146e1f94 WatchSource:0}: Error finding container 449dedc0ec25986c870defdf07c4124f76d72711164ed82b3baaabed146e1f94: Status 404 returned error can't find the container with id 449dedc0ec25986c870defdf07c4124f76d72711164ed82b3baaabed146e1f94 Nov 03 17:09:52.260161 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:52.260134 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6"] Nov 03 17:09:52.262009 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:52.261976 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:09:52.262078 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:52.262045 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:09:52.262078 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:52.262072 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:09:53.036550 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:53.036509 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" event={"ID":"e7c02310-bff9-4d86-b641-8022d891b395","Type":"ContainerStarted","Data":"25ecc4cc8c564e3de6e8805bd72df40a63354305526c3413dcad51df711ac170"} Nov 03 17:09:53.037050 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:53.036575 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" event={"ID":"e7c02310-bff9-4d86-b641-8022d891b395","Type":"ContainerStarted","Data":"449dedc0ec25986c870defdf07c4124f76d72711164ed82b3baaabed146e1f94"} Nov 03 17:09:53.123896 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:53.123828 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" podStartSLOduration=2.123809836 podStartE2EDuration="2.123809836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:09:53.114757749 +0000 UTC m=+3336.703757719" watchObservedRunningTime="2026-11-03 17:09:53.123809836 +0000 UTC m=+3336.712809806" Nov 03 17:09:53.926932 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:53.926892 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:53.929004 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:53.928975 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 17:09:53.929134 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:53.929023 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" podUID="e7c02310-bff9-4d86-b641-8022d891b395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 17:09:53.974834 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:53.974779 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" podUID="9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa" containerName="payload-processing" containerID="cri-o://1836591c247f58ea8d0e452f91d0883ebcbab6bfc5fb096d4dbe66e7cab06aa2" gracePeriod=25 Nov 03 17:09:53.985613 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:53.985570 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" podUID="e61163db-7a96-4fba-b15c-956a8cabe4de" containerName="payload-processing" containerID="cri-o://ea37ed5c2a16268ddba14d712984493821b74b68b03733621664f572aa708134" gracePeriod=25 Nov 03 17:09:54.352377 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.352347 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" Nov 03 17:09:54.444537 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.443234 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" Nov 03 17:09:54.495123 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.495082 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-plugins-config-volume\") pod \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\" (UID: \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\") " Nov 03 17:09:54.495342 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.495281 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-routing-overlay-volume\") pod \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\" (UID: \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\") " Nov 03 17:09:54.495424 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.495340 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-kube-api-access-tpg2z\" (UniqueName: \"kubernetes.io/projected/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-kube-api-access-tpg2z\") pod \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\" (UID: \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\") " Nov 03 17:09:54.496200 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.495529 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-provider-credentials\") pod \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\" (UID: \"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa\") " Nov 03 17:09:54.496200 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.495737 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-routing-overlay-volume" pod "9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa" (UID: "9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:54.496200 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.495960 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-plugins-config-volume" pod "9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa" (UID: "9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:54.496659 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.496636 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:54.496784 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.496664 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:54.533889 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.532458 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-kube-api-access-tpg2z" pod "9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa" (UID: "9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa"). InnerVolumeSpecName "kube-api-access-tpg2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:54.534547 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.534513 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-provider-credentials" pod "9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa" (UID: "9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:54.597470 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.597438 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e61163db-7a96-4fba-b15c-956a8cabe4de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e61163db-7a96-4fba-b15c-956a8cabe4de-plugins-config-volume\") pod \"e61163db-7a96-4fba-b15c-956a8cabe4de\" (UID: \"e61163db-7a96-4fba-b15c-956a8cabe4de\") " Nov 03 17:09:54.597647 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.597541 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e61163db-7a96-4fba-b15c-956a8cabe4de-kube-api-access-xff27\" (UniqueName: \"kubernetes.io/projected/e61163db-7a96-4fba-b15c-956a8cabe4de-kube-api-access-xff27\") pod \"e61163db-7a96-4fba-b15c-956a8cabe4de\" (UID: \"e61163db-7a96-4fba-b15c-956a8cabe4de\") " Nov 03 17:09:54.597647 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.597578 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e61163db-7a96-4fba-b15c-956a8cabe4de-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e61163db-7a96-4fba-b15c-956a8cabe4de-routing-overlay-volume\") pod \"e61163db-7a96-4fba-b15c-956a8cabe4de\" (UID: \"e61163db-7a96-4fba-b15c-956a8cabe4de\") " Nov 03 17:09:54.597833 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.597667 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e61163db-7a96-4fba-b15c-956a8cabe4de-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e61163db-7a96-4fba-b15c-956a8cabe4de-provider-credentials\") pod \"e61163db-7a96-4fba-b15c-956a8cabe4de\" (UID: \"e61163db-7a96-4fba-b15c-956a8cabe4de\") " Nov 03 17:09:54.605723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.598569 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpg2z\" (UniqueName: \"kubernetes.io/projected/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-kube-api-access-tpg2z\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:54.605723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.598601 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:54.605723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.602973 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61163db-7a96-4fba-b15c-956a8cabe4de-plugins-config-volume" pod "e61163db-7a96-4fba-b15c-956a8cabe4de" (UID: "e61163db-7a96-4fba-b15c-956a8cabe4de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:54.605723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.603320 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61163db-7a96-4fba-b15c-956a8cabe4de-routing-overlay-volume" pod "e61163db-7a96-4fba-b15c-956a8cabe4de" (UID: "e61163db-7a96-4fba-b15c-956a8cabe4de"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:09:54.629927 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.629881 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61163db-7a96-4fba-b15c-956a8cabe4de-provider-credentials" pod "e61163db-7a96-4fba-b15c-956a8cabe4de" (UID: "e61163db-7a96-4fba-b15c-956a8cabe4de"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:54.630161 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.630040 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61163db-7a96-4fba-b15c-956a8cabe4de-kube-api-access-xff27" pod "e61163db-7a96-4fba-b15c-956a8cabe4de" (UID: "e61163db-7a96-4fba-b15c-956a8cabe4de"). InnerVolumeSpecName "kube-api-access-xff27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:09:54.719052 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.709600 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e61163db-7a96-4fba-b15c-956a8cabe4de-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:54.719317 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.719291 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e61163db-7a96-4fba-b15c-956a8cabe4de-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:54.719529 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.719481 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xff27\" (UniqueName: \"kubernetes.io/projected/e61163db-7a96-4fba-b15c-956a8cabe4de-kube-api-access-xff27\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:54.719643 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.719630 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e61163db-7a96-4fba-b15c-956a8cabe4de-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:09:54.931772 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.927745 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 17:09:54.931772 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:54.927823 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" podUID="e7c02310-bff9-4d86-b641-8022d891b395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 17:09:55.155747 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.155687 4947 generic.go:356] "Generic (PLEG): container finished" podID="e61163db-7a96-4fba-b15c-956a8cabe4de" containerID="ea37ed5c2a16268ddba14d712984493821b74b68b03733621664f572aa708134" exitCode=0 Nov 03 17:09:55.156388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.156243 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" event={"ID":"e61163db-7a96-4fba-b15c-956a8cabe4de","Type":"ContainerDied","Data":"ea37ed5c2a16268ddba14d712984493821b74b68b03733621664f572aa708134"} Nov 03 17:09:55.156591 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.156570 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" event={"ID":"e61163db-7a96-4fba-b15c-956a8cabe4de","Type":"ContainerDied","Data":"9d7a3e98525248739f11c622252d41b1cfbf548e22b6f50c3dc1411541b8775b"} Nov 03 17:09:55.156739 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.156726 4947 scope.go:122] "RemoveContainer" containerID="ea37ed5c2a16268ddba14d712984493821b74b68b03733621664f572aa708134" Nov 03 17:09:55.157007 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.156992 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx" Nov 03 17:09:55.168473 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.168450 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" event={"ID":"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa","Type":"ContainerDied","Data":"1836591c247f58ea8d0e452f91d0883ebcbab6bfc5fb096d4dbe66e7cab06aa2"} Nov 03 17:09:55.168897 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.168878 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" Nov 03 17:09:55.169185 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.168322 4947 generic.go:356] "Generic (PLEG): container finished" podID="9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa" containerID="1836591c247f58ea8d0e452f91d0883ebcbab6bfc5fb096d4dbe66e7cab06aa2" exitCode=0 Nov 03 17:09:55.169390 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.169367 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh" event={"ID":"9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa","Type":"ContainerDied","Data":"c7c06faf64bf603b072df0903b5d2fb4c9df244fab2342785f47590e104b51b2"} Nov 03 17:09:55.226938 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.226909 4947 scope.go:122] "RemoveContainer" containerID="ea37ed5c2a16268ddba14d712984493821b74b68b03733621664f572aa708134" Nov 03 17:09:55.247195 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:55.247148 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea37ed5c2a16268ddba14d712984493821b74b68b03733621664f572aa708134\": container with ID starting with ea37ed5c2a16268ddba14d712984493821b74b68b03733621664f572aa708134 not found: ID does not exist" containerID="ea37ed5c2a16268ddba14d712984493821b74b68b03733621664f572aa708134" Nov 03 17:09:55.247503 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.247450 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea37ed5c2a16268ddba14d712984493821b74b68b03733621664f572aa708134"} err="failed to get container status \"ea37ed5c2a16268ddba14d712984493821b74b68b03733621664f572aa708134\": rpc error: code = NotFound desc = could not find container \"ea37ed5c2a16268ddba14d712984493821b74b68b03733621664f572aa708134\": container with ID starting with ea37ed5c2a16268ddba14d712984493821b74b68b03733621664f572aa708134 not found: ID does not exist" Nov 03 17:09:55.247655 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.247641 4947 scope.go:122] "RemoveContainer" containerID="1836591c247f58ea8d0e452f91d0883ebcbab6bfc5fb096d4dbe66e7cab06aa2" Nov 03 17:09:55.273883 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.260752 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh"] Nov 03 17:09:55.308829 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.308787 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-7677sxsnh"] Nov 03 17:09:55.404512 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.404383 4947 scope.go:122] "RemoveContainer" containerID="1836591c247f58ea8d0e452f91d0883ebcbab6bfc5fb096d4dbe66e7cab06aa2" Nov 03 17:09:55.405895 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:09:55.405664 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1836591c247f58ea8d0e452f91d0883ebcbab6bfc5fb096d4dbe66e7cab06aa2\": container with ID starting with 1836591c247f58ea8d0e452f91d0883ebcbab6bfc5fb096d4dbe66e7cab06aa2 not found: ID does not exist" containerID="1836591c247f58ea8d0e452f91d0883ebcbab6bfc5fb096d4dbe66e7cab06aa2" Nov 03 17:09:55.405895 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.405727 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1836591c247f58ea8d0e452f91d0883ebcbab6bfc5fb096d4dbe66e7cab06aa2"} err="failed to get container status \"1836591c247f58ea8d0e452f91d0883ebcbab6bfc5fb096d4dbe66e7cab06aa2\": rpc error: code = NotFound desc = could not find container \"1836591c247f58ea8d0e452f91d0883ebcbab6bfc5fb096d4dbe66e7cab06aa2\": container with ID starting with 1836591c247f58ea8d0e452f91d0883ebcbab6bfc5fb096d4dbe66e7cab06aa2 not found: ID does not exist" Nov 03 17:09:55.405895 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.405774 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx"] Nov 03 17:09:55.460666 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.460636 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-c4d4vfqvx"] Nov 03 17:09:55.787737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.786451 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:09:55.886723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.885771 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:09:55.927725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.927077 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 17:09:55.927725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:55.927151 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" podUID="e7c02310-bff9-4d86-b641-8022d891b395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 17:09:56.568605 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:56.568571 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa" path="/var/lib/kubelet/pods/9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa/volumes" Nov 03 17:09:56.569221 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:56.569195 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e61163db-7a96-4fba-b15c-956a8cabe4de" path="/var/lib/kubelet/pods/e61163db-7a96-4fba-b15c-956a8cabe4de/volumes" Nov 03 17:09:56.933779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:56.926997 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 17:09:56.933779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:56.927092 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" podUID="e7c02310-bff9-4d86-b641-8022d891b395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 17:09:57.928724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:57.928217 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 17:09:57.928724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:57.928280 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" podUID="e7c02310-bff9-4d86-b641-8022d891b395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 17:09:57.981055 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:57.980111 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" Nov 03 17:09:57.981055 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:57.980177 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" Nov 03 17:09:58.150867 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:58.150827 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt"] Nov 03 17:09:58.946727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:58.936181 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Nov 03 17:09:58.946727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:58.936266 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" podUID="e7c02310-bff9-4d86-b641-8022d891b395" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Nov 03 17:09:59.932007 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:59.931974 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:59.932399 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:59.932368 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:09:59.933110 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:09:59.933091 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:10:00.132417 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:00.132378 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:10:03.199970 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:03.199925 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" podUID="e5daa154-497b-4bfe-98ce-1c0af39b0224" containerName="payload-processing" containerID="cri-o://76bef5801000024a1126b6d65fed2216b5c7c227dc615789690ff6c9423040b1" gracePeriod=25 Nov 03 17:10:03.495874 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:03.495842 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" Nov 03 17:10:03.593683 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:03.593638 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5daa154-497b-4bfe-98ce-1c0af39b0224-kube-api-access-c4b24\" (UniqueName: \"kubernetes.io/projected/e5daa154-497b-4bfe-98ce-1c0af39b0224-kube-api-access-c4b24\") pod \"e5daa154-497b-4bfe-98ce-1c0af39b0224\" (UID: \"e5daa154-497b-4bfe-98ce-1c0af39b0224\") " Nov 03 17:10:03.593910 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:03.593832 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5daa154-497b-4bfe-98ce-1c0af39b0224-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5daa154-497b-4bfe-98ce-1c0af39b0224-routing-overlay-volume\") pod \"e5daa154-497b-4bfe-98ce-1c0af39b0224\" (UID: \"e5daa154-497b-4bfe-98ce-1c0af39b0224\") " Nov 03 17:10:03.594065 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:03.594047 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5daa154-497b-4bfe-98ce-1c0af39b0224-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5daa154-497b-4bfe-98ce-1c0af39b0224-provider-credentials\") pod \"e5daa154-497b-4bfe-98ce-1c0af39b0224\" (UID: \"e5daa154-497b-4bfe-98ce-1c0af39b0224\") " Nov 03 17:10:03.594207 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:03.594174 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5daa154-497b-4bfe-98ce-1c0af39b0224-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5daa154-497b-4bfe-98ce-1c0af39b0224-plugins-config-volume\") pod \"e5daa154-497b-4bfe-98ce-1c0af39b0224\" (UID: \"e5daa154-497b-4bfe-98ce-1c0af39b0224\") " Nov 03 17:10:03.595466 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:03.594301 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5daa154-497b-4bfe-98ce-1c0af39b0224-routing-overlay-volume" pod "e5daa154-497b-4bfe-98ce-1c0af39b0224" (UID: "e5daa154-497b-4bfe-98ce-1c0af39b0224"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:03.595466 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:03.594645 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5daa154-497b-4bfe-98ce-1c0af39b0224-plugins-config-volume" pod "e5daa154-497b-4bfe-98ce-1c0af39b0224" (UID: "e5daa154-497b-4bfe-98ce-1c0af39b0224"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:03.595466 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:03.595009 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5daa154-497b-4bfe-98ce-1c0af39b0224-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:03.595466 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:03.595031 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e5daa154-497b-4bfe-98ce-1c0af39b0224-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:03.597357 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:03.597322 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5daa154-497b-4bfe-98ce-1c0af39b0224-provider-credentials" pod "e5daa154-497b-4bfe-98ce-1c0af39b0224" (UID: "e5daa154-497b-4bfe-98ce-1c0af39b0224"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:03.597474 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:03.597403 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5daa154-497b-4bfe-98ce-1c0af39b0224-kube-api-access-c4b24" pod "e5daa154-497b-4bfe-98ce-1c0af39b0224" (UID: "e5daa154-497b-4bfe-98ce-1c0af39b0224"). InnerVolumeSpecName "kube-api-access-c4b24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:03.698617 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:03.698373 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e5daa154-497b-4bfe-98ce-1c0af39b0224-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:03.698617 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:03.698413 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4b24\" (UniqueName: \"kubernetes.io/projected/e5daa154-497b-4bfe-98ce-1c0af39b0224-kube-api-access-c4b24\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:04.253787 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:04.253753 4947 generic.go:356] "Generic (PLEG): container finished" podID="e5daa154-497b-4bfe-98ce-1c0af39b0224" containerID="76bef5801000024a1126b6d65fed2216b5c7c227dc615789690ff6c9423040b1" exitCode=0 Nov 03 17:10:04.253787 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:04.253784 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" event={"ID":"e5daa154-497b-4bfe-98ce-1c0af39b0224","Type":"ContainerDied","Data":"76bef5801000024a1126b6d65fed2216b5c7c227dc615789690ff6c9423040b1"} Nov 03 17:10:04.254356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:04.253828 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" event={"ID":"e5daa154-497b-4bfe-98ce-1c0af39b0224","Type":"ContainerDied","Data":"a6e12248d613b140ffcfe7933c5f11a041b37ac15881eed21111a11bd7e9fbae"} Nov 03 17:10:04.254356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:04.253850 4947 scope.go:122] "RemoveContainer" containerID="76bef5801000024a1126b6d65fed2216b5c7c227dc615789690ff6c9423040b1" Nov 03 17:10:04.254356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:04.253767 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt" Nov 03 17:10:04.266050 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:04.266028 4947 scope.go:122] "RemoveContainer" containerID="76bef5801000024a1126b6d65fed2216b5c7c227dc615789690ff6c9423040b1" Nov 03 17:10:04.266373 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:04.266353 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76bef5801000024a1126b6d65fed2216b5c7c227dc615789690ff6c9423040b1\": container with ID starting with 76bef5801000024a1126b6d65fed2216b5c7c227dc615789690ff6c9423040b1 not found: ID does not exist" containerID="76bef5801000024a1126b6d65fed2216b5c7c227dc615789690ff6c9423040b1" Nov 03 17:10:04.266462 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:04.266380 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76bef5801000024a1126b6d65fed2216b5c7c227dc615789690ff6c9423040b1"} err="failed to get container status \"76bef5801000024a1126b6d65fed2216b5c7c227dc615789690ff6c9423040b1\": rpc error: code = NotFound desc = could not find container \"76bef5801000024a1126b6d65fed2216b5c7c227dc615789690ff6c9423040b1\": container with ID starting with 76bef5801000024a1126b6d65fed2216b5c7c227dc615789690ff6c9423040b1 not found: ID does not exist" Nov 03 17:10:04.328343 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:04.328306 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt"] Nov 03 17:10:04.341790 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:04.341754 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-59558gmbt"] Nov 03 17:10:04.568214 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:04.568115 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5daa154-497b-4bfe-98ce-1c0af39b0224" path="/var/lib/kubelet/pods/e5daa154-497b-4bfe-98ce-1c0af39b0224/volumes" Nov 03 17:10:06.570268 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.570240 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw"] Nov 03 17:10:06.571006 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.570991 4947 cpu_manager.go:414] "containerMap: removing container" podUID="9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa" containerName="payload-processing" Nov 03 17:10:06.571062 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.571008 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa" containerName="payload-processing" Nov 03 17:10:06.571062 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.571028 4947 cpu_manager.go:414] "containerMap: removing container" podUID="e61163db-7a96-4fba-b15c-956a8cabe4de" containerName="payload-processing" Nov 03 17:10:06.571062 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.571033 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e61163db-7a96-4fba-b15c-956a8cabe4de" containerName="payload-processing" Nov 03 17:10:06.571062 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.571044 4947 cpu_manager.go:414] "containerMap: removing container" podUID="e5daa154-497b-4bfe-98ce-1c0af39b0224" containerName="payload-processing" Nov 03 17:10:06.571062 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.571048 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5daa154-497b-4bfe-98ce-1c0af39b0224" containerName="payload-processing" Nov 03 17:10:06.571233 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.571187 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbeaf8b-c161-4a9d-91ab-eeed04ac03fa" containerName="payload-processing" Nov 03 17:10:06.571233 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.571196 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e61163db-7a96-4fba-b15c-956a8cabe4de" containerName="payload-processing" Nov 03 17:10:06.571233 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.571204 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5daa154-497b-4bfe-98ce-1c0af39b0224" containerName="payload-processing" Nov 03 17:10:06.576771 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.576743 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" Nov 03 17:10:06.590107 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.590072 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw"] Nov 03 17:10:06.600058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.599140 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"routing-overlay\"" Nov 03 17:10:06.600058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.599309 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"agc-cred-c299a4d1286c\"" Nov 03 17:10:06.600058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.599337 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-da409b\"" Nov 03 17:10:06.600058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.599367 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"openshift-service-ca.crt\"" Nov 03 17:10:06.600058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.599399 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"kube-root-ca.crt\"" Nov 03 17:10:06.607571 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.607534 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"payload-processing-external-model-e2e--f0537159-dockercfg-kc6pc\"" Nov 03 17:10:06.662791 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.662739 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/32836a71-8b45-4d9c-be6a-995afe63f9df-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw\" (UID: \"32836a71-8b45-4d9c-be6a-995afe63f9df\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" Nov 03 17:10:06.663000 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.662939 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgkgd\" (UniqueName: \"kubernetes.io/projected/32836a71-8b45-4d9c-be6a-995afe63f9df-kube-api-access-qgkgd\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw\" (UID: \"32836a71-8b45-4d9c-be6a-995afe63f9df\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" Nov 03 17:10:06.663268 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.663249 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32836a71-8b45-4d9c-be6a-995afe63f9df-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw\" (UID: \"32836a71-8b45-4d9c-be6a-995afe63f9df\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" Nov 03 17:10:06.663339 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.663321 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/32836a71-8b45-4d9c-be6a-995afe63f9df-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw\" (UID: \"32836a71-8b45-4d9c-be6a-995afe63f9df\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" Nov 03 17:10:06.802726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.793164 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/32836a71-8b45-4d9c-be6a-995afe63f9df-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw\" (UID: \"32836a71-8b45-4d9c-be6a-995afe63f9df\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" Nov 03 17:10:06.802726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.793249 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/32836a71-8b45-4d9c-be6a-995afe63f9df-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw\" (UID: \"32836a71-8b45-4d9c-be6a-995afe63f9df\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" Nov 03 17:10:06.802726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.793361 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgkgd\" (UniqueName: \"kubernetes.io/projected/32836a71-8b45-4d9c-be6a-995afe63f9df-kube-api-access-qgkgd\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw\" (UID: \"32836a71-8b45-4d9c-be6a-995afe63f9df\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" Nov 03 17:10:06.802726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.793609 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32836a71-8b45-4d9c-be6a-995afe63f9df-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw\" (UID: \"32836a71-8b45-4d9c-be6a-995afe63f9df\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" Nov 03 17:10:06.802726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.794624 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32836a71-8b45-4d9c-be6a-995afe63f9df-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw\" (UID: \"32836a71-8b45-4d9c-be6a-995afe63f9df\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" Nov 03 17:10:06.802726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.795246 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/32836a71-8b45-4d9c-be6a-995afe63f9df-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw\" (UID: \"32836a71-8b45-4d9c-be6a-995afe63f9df\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" Nov 03 17:10:06.845946 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.845828 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/32836a71-8b45-4d9c-be6a-995afe63f9df-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw\" (UID: \"32836a71-8b45-4d9c-be6a-995afe63f9df\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" Nov 03 17:10:06.869105 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.869036 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgkgd\" (UniqueName: \"kubernetes.io/projected/32836a71-8b45-4d9c-be6a-995afe63f9df-kube-api-access-qgkgd\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw\" (UID: \"32836a71-8b45-4d9c-be6a-995afe63f9df\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" Nov 03 17:10:06.897176 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:06.897140 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" Nov 03 17:10:07.266641 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:07.266600 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw"] Nov 03 17:10:07.314891 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:07.314852 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" event={"ID":"32836a71-8b45-4d9c-be6a-995afe63f9df","Type":"ContainerStarted","Data":"0050cb4c7ad8dac602cb43fa23cb26b6392a5a318aa213f8afd9afb843d8f5ea"} Nov 03 17:10:08.323764 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:08.323689 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" event={"ID":"32836a71-8b45-4d9c-be6a-995afe63f9df","Type":"ContainerStarted","Data":"b6aa6880b8572ede6b388d660b3155ca3e7b6d722c563ee4fdd439f3be932c75"} Nov 03 17:10:08.324326 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:08.323803 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" Nov 03 17:10:08.371498 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:08.371434 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" podStartSLOduration=2.371418772 podStartE2EDuration="2.371418772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:08.368216482 +0000 UTC m=+3351.957216451" watchObservedRunningTime="2026-11-03 17:10:08.371418772 +0000 UTC m=+3351.960418741" Nov 03 17:10:13.108054 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:13.108011 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md"] Nov 03 17:10:15.319548 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:15.319514 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z"] Nov 03 17:10:15.327336 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:15.327288 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" Nov 03 17:10:15.399730 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:15.394015 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z"] Nov 03 17:10:15.501032 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:15.500144 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z\" (UID: \"609f8f4e-e735-4055-9712-ae87b32bbe51\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" Nov 03 17:10:15.501032 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:15.500259 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/609f8f4e-e735-4055-9712-ae87b32bbe51-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z\" (UID: \"609f8f4e-e735-4055-9712-ae87b32bbe51\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" Nov 03 17:10:15.501032 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:15.500430 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609f8f4e-e735-4055-9712-ae87b32bbe51-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z\" (UID: \"609f8f4e-e735-4055-9712-ae87b32bbe51\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" Nov 03 17:10:15.501032 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:15.500458 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8wmc\" (UniqueName: \"kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-kube-api-access-g8wmc\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z\" (UID: \"609f8f4e-e735-4055-9712-ae87b32bbe51\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" Nov 03 17:10:15.602828 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:15.602680 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z\" (UID: \"609f8f4e-e735-4055-9712-ae87b32bbe51\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" Nov 03 17:10:15.602828 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:15.602810 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/609f8f4e-e735-4055-9712-ae87b32bbe51-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z\" (UID: \"609f8f4e-e735-4055-9712-ae87b32bbe51\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" Nov 03 17:10:15.603085 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:15.602960 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609f8f4e-e735-4055-9712-ae87b32bbe51-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z\" (UID: \"609f8f4e-e735-4055-9712-ae87b32bbe51\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" Nov 03 17:10:15.603085 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:15.602993 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8wmc\" (UniqueName: \"kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-kube-api-access-g8wmc\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z\" (UID: \"609f8f4e-e735-4055-9712-ae87b32bbe51\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" Nov 03 17:10:15.603634 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:15.603605 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/609f8f4e-e735-4055-9712-ae87b32bbe51-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z\" (UID: \"609f8f4e-e735-4055-9712-ae87b32bbe51\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" Nov 03 17:10:15.603865 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:15.603841 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609f8f4e-e735-4055-9712-ae87b32bbe51-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z\" (UID: \"609f8f4e-e735-4055-9712-ae87b32bbe51\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" Nov 03 17:10:15.605897 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:15.605876 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z\" (UID: \"609f8f4e-e735-4055-9712-ae87b32bbe51\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" Nov 03 17:10:15.628279 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:15.628221 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8wmc\" (UniqueName: \"kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-kube-api-access-g8wmc\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z\" (UID: \"609f8f4e-e735-4055-9712-ae87b32bbe51\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" Nov 03 17:10:15.823532 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:15.823010 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" Nov 03 17:10:15.911039 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:15.873519 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682"] Nov 03 17:10:16.100562 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.099907 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.100958 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.100939 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682"] Nov 03 17:10:16.141858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.141825 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.142053 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.141883 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.142137 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.142070 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.142137 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.142127 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.142259 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.142223 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.142317 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.142280 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.142411 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.142359 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfmkn\" (UniqueName: \"kubernetes.io/projected/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-kube-api-access-xfmkn\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.142537 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.142521 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.142748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.142598 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.243994 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.243956 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.244213 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.244014 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.244213 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.244092 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfmkn\" (UniqueName: \"kubernetes.io/projected/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-kube-api-access-xfmkn\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.244213 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.244179 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.244403 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.244255 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.244403 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.244352 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.244403 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.244392 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.244569 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.244426 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.244569 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.244537 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.244686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.244593 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.244686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.244675 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.244979 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.244956 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.245050 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.244982 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.245549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.245452 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.247611 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.247585 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.247761 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.247725 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.269825 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.269791 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.270254 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.270226 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfmkn\" (UniqueName: \"kubernetes.io/projected/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-kube-api-access-xfmkn\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-l2682\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.414629 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.414593 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z"] Nov 03 17:10:16.428489 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.428455 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:16.536061 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.536031 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" event={"ID":"609f8f4e-e735-4055-9712-ae87b32bbe51","Type":"ContainerStarted","Data":"0f99295e477a52e43d6636c99d8fb481436b0c0db897e90df33ec2a503a97b90"} Nov 03 17:10:16.536281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.536266 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" event={"ID":"609f8f4e-e735-4055-9712-ae87b32bbe51","Type":"ContainerStarted","Data":"2333c99007cdd0bce64333680f5ec97c3a0db90c9fcc1e73f50946da8221fa9b"} Nov 03 17:10:16.538024 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.537063 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" Nov 03 17:10:16.600845 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.600775 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" podStartSLOduration=1.6007484619999999 podStartE2EDuration="1.600748462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:16.59748931 +0000 UTC m=+3360.186489280" watchObservedRunningTime="2026-11-03 17:10:16.600748462 +0000 UTC m=+3360.189748433" Nov 03 17:10:16.820649 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.820584 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:10:16.822318 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.822282 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:10:16.822468 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.822347 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:10:16.822789 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.821297 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682"] Nov 03 17:10:16.826293 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.826270 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-559b8977dd-97dg9"] Nov 03 17:10:16.832945 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.832920 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" Nov 03 17:10:16.859208 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.859182 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-lfpl8\"" Nov 03 17:10:16.861472 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.861450 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d54f748cefbc\"" Nov 03 17:10:16.869643 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.869610 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxxzv\" (UniqueName: \"kubernetes.io/projected/4332898c-f56e-4ab6-b099-988456a34bf7-kube-api-access-gxxzv\") pod \"payload-processing-external-model-559b8977dd-97dg9\" (UID: \"4332898c-f56e-4ab6-b099-988456a34bf7\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" Nov 03 17:10:16.869883 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.869862 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4332898c-f56e-4ab6-b099-988456a34bf7-routing-overlay-volume\") pod \"payload-processing-external-model-559b8977dd-97dg9\" (UID: \"4332898c-f56e-4ab6-b099-988456a34bf7\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" Nov 03 17:10:16.870045 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.870030 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4332898c-f56e-4ab6-b099-988456a34bf7-plugins-config-volume\") pod \"payload-processing-external-model-559b8977dd-97dg9\" (UID: \"4332898c-f56e-4ab6-b099-988456a34bf7\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" Nov 03 17:10:16.870164 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.870143 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4332898c-f56e-4ab6-b099-988456a34bf7-provider-credentials\") pod \"payload-processing-external-model-559b8977dd-97dg9\" (UID: \"4332898c-f56e-4ab6-b099-988456a34bf7\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" Nov 03 17:10:16.876292 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.876260 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-559b8977dd-97dg9"] Nov 03 17:10:16.972520 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.972479 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4332898c-f56e-4ab6-b099-988456a34bf7-plugins-config-volume\") pod \"payload-processing-external-model-559b8977dd-97dg9\" (UID: \"4332898c-f56e-4ab6-b099-988456a34bf7\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" Nov 03 17:10:16.972870 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.972846 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4332898c-f56e-4ab6-b099-988456a34bf7-provider-credentials\") pod \"payload-processing-external-model-559b8977dd-97dg9\" (UID: \"4332898c-f56e-4ab6-b099-988456a34bf7\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" Nov 03 17:10:16.973182 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.973162 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxxzv\" (UniqueName: \"kubernetes.io/projected/4332898c-f56e-4ab6-b099-988456a34bf7-kube-api-access-gxxzv\") pod \"payload-processing-external-model-559b8977dd-97dg9\" (UID: \"4332898c-f56e-4ab6-b099-988456a34bf7\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" Nov 03 17:10:16.973478 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.973455 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4332898c-f56e-4ab6-b099-988456a34bf7-routing-overlay-volume\") pod \"payload-processing-external-model-559b8977dd-97dg9\" (UID: \"4332898c-f56e-4ab6-b099-988456a34bf7\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" Nov 03 17:10:16.974164 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.974134 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4332898c-f56e-4ab6-b099-988456a34bf7-routing-overlay-volume\") pod \"payload-processing-external-model-559b8977dd-97dg9\" (UID: \"4332898c-f56e-4ab6-b099-988456a34bf7\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" Nov 03 17:10:16.974828 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.974804 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4332898c-f56e-4ab6-b099-988456a34bf7-plugins-config-volume\") pod \"payload-processing-external-model-559b8977dd-97dg9\" (UID: \"4332898c-f56e-4ab6-b099-988456a34bf7\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" Nov 03 17:10:16.977806 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.977781 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4332898c-f56e-4ab6-b099-988456a34bf7-provider-credentials\") pod \"payload-processing-external-model-559b8977dd-97dg9\" (UID: \"4332898c-f56e-4ab6-b099-988456a34bf7\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" Nov 03 17:10:16.998580 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:16.998549 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxxzv\" (UniqueName: \"kubernetes.io/projected/4332898c-f56e-4ab6-b099-988456a34bf7-kube-api-access-gxxzv\") pod \"payload-processing-external-model-559b8977dd-97dg9\" (UID: \"4332898c-f56e-4ab6-b099-988456a34bf7\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" Nov 03 17:10:17.169772 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:17.167141 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" Nov 03 17:10:17.547025 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:17.546988 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" event={"ID":"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009","Type":"ContainerStarted","Data":"4292063619ab9038f48bf110d15c1f918027deaff9231717a14404efd2692760"} Nov 03 17:10:17.547523 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:17.547045 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" event={"ID":"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009","Type":"ContainerStarted","Data":"be40fca6fe680d9fcb8665c406a5223421c73b7e1a5d2ca8f46568efa2b35ac0"} Nov 03 17:10:17.638191 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:17.638124 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" podStartSLOduration=2.638104873 podStartE2EDuration="2.638104873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:17.636676058 +0000 UTC m=+3361.225676029" watchObservedRunningTime="2026-11-03 17:10:17.638104873 +0000 UTC m=+3361.227104843" Nov 03 17:10:17.656224 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:10:17.656186 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4332898c_f56e_4ab6_b099_988456a34bf7.slice/crio-4f594d37c72adbf912ab4a8d7c336e0c2141285cce4bfbbb9d8968555f703035 WatchSource:0}: Error finding container 4f594d37c72adbf912ab4a8d7c336e0c2141285cce4bfbbb9d8968555f703035: Status 404 returned error can't find the container with id 4f594d37c72adbf912ab4a8d7c336e0c2141285cce4bfbbb9d8968555f703035 Nov 03 17:10:17.656530 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:17.656503 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-559b8977dd-97dg9"] Nov 03 17:10:18.118862 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.118820 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" podUID="df29aa6b-58f4-48b9-87b6-09b6b50260e6" containerName="payload-processing" containerID="cri-o://faa84332b93fc6bc1cc5224940c853c16b38e0aec940a48a443b8559bc928454" gracePeriod=25 Nov 03 17:10:18.429466 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.429385 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:18.431116 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.431088 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-l2682 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 03 17:10:18.431229 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.431141 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" podUID="629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 03 17:10:18.483274 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.483248 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" Nov 03 17:10:18.573351 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.573317 4947 generic.go:356] "Generic (PLEG): container finished" podID="df29aa6b-58f4-48b9-87b6-09b6b50260e6" containerID="faa84332b93fc6bc1cc5224940c853c16b38e0aec940a48a443b8559bc928454" exitCode=0 Nov 03 17:10:18.574299 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.574274 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" Nov 03 17:10:18.578829 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.578810 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" Nov 03 17:10:18.578973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.578857 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" event={"ID":"4332898c-f56e-4ab6-b099-988456a34bf7","Type":"ContainerStarted","Data":"43fb203fb5f89fc78b2717af235d9a415f2ae4eb05ecdbbc37fed5a8539ce1b2"} Nov 03 17:10:18.578973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.578887 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" event={"ID":"4332898c-f56e-4ab6-b099-988456a34bf7","Type":"ContainerStarted","Data":"4f594d37c72adbf912ab4a8d7c336e0c2141285cce4bfbbb9d8968555f703035"} Nov 03 17:10:18.578973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.578912 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" event={"ID":"df29aa6b-58f4-48b9-87b6-09b6b50260e6","Type":"ContainerDied","Data":"faa84332b93fc6bc1cc5224940c853c16b38e0aec940a48a443b8559bc928454"} Nov 03 17:10:18.578973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.578938 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md" event={"ID":"df29aa6b-58f4-48b9-87b6-09b6b50260e6","Type":"ContainerDied","Data":"855b1b0bdb5f77a1d43c5763dc03a93450f92b684ca87705b597bba7c4720427"} Nov 03 17:10:18.578973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.578970 4947 scope.go:122] "RemoveContainer" containerID="faa84332b93fc6bc1cc5224940c853c16b38e0aec940a48a443b8559bc928454" Nov 03 17:10:18.590571 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.590550 4947 scope.go:122] "RemoveContainer" containerID="faa84332b93fc6bc1cc5224940c853c16b38e0aec940a48a443b8559bc928454" Nov 03 17:10:18.590933 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:18.590908 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa84332b93fc6bc1cc5224940c853c16b38e0aec940a48a443b8559bc928454\": container with ID starting with faa84332b93fc6bc1cc5224940c853c16b38e0aec940a48a443b8559bc928454 not found: ID does not exist" containerID="faa84332b93fc6bc1cc5224940c853c16b38e0aec940a48a443b8559bc928454" Nov 03 17:10:18.591058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.590942 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa84332b93fc6bc1cc5224940c853c16b38e0aec940a48a443b8559bc928454"} err="failed to get container status \"faa84332b93fc6bc1cc5224940c853c16b38e0aec940a48a443b8559bc928454\": rpc error: code = NotFound desc = could not find container \"faa84332b93fc6bc1cc5224940c853c16b38e0aec940a48a443b8559bc928454\": container with ID starting with faa84332b93fc6bc1cc5224940c853c16b38e0aec940a48a443b8559bc928454 not found: ID does not exist" Nov 03 17:10:18.638133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.638013 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" podStartSLOduration=2.637989996 podStartE2EDuration="2.637989996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:18.624021636 +0000 UTC m=+3362.213021606" watchObservedRunningTime="2026-11-03 17:10:18.637989996 +0000 UTC m=+3362.226989966" Nov 03 17:10:18.664605 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.664559 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df29aa6b-58f4-48b9-87b6-09b6b50260e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df29aa6b-58f4-48b9-87b6-09b6b50260e6-plugins-config-volume\") pod \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\" (UID: \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\") " Nov 03 17:10:18.664924 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.664903 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df29aa6b-58f4-48b9-87b6-09b6b50260e6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df29aa6b-58f4-48b9-87b6-09b6b50260e6-routing-overlay-volume\") pod \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\" (UID: \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\") " Nov 03 17:10:18.664982 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.664948 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df29aa6b-58f4-48b9-87b6-09b6b50260e6-kube-api-access-mpp6c\" (UniqueName: \"kubernetes.io/projected/df29aa6b-58f4-48b9-87b6-09b6b50260e6-kube-api-access-mpp6c\") pod \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\" (UID: \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\") " Nov 03 17:10:18.665118 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.665102 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df29aa6b-58f4-48b9-87b6-09b6b50260e6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/df29aa6b-58f4-48b9-87b6-09b6b50260e6-provider-credentials\") pod \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\" (UID: \"df29aa6b-58f4-48b9-87b6-09b6b50260e6\") " Nov 03 17:10:18.665336 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.665310 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df29aa6b-58f4-48b9-87b6-09b6b50260e6-routing-overlay-volume" pod "df29aa6b-58f4-48b9-87b6-09b6b50260e6" (UID: "df29aa6b-58f4-48b9-87b6-09b6b50260e6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:18.665938 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.665910 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df29aa6b-58f4-48b9-87b6-09b6b50260e6-plugins-config-volume" pod "df29aa6b-58f4-48b9-87b6-09b6b50260e6" (UID: "df29aa6b-58f4-48b9-87b6-09b6b50260e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:18.666567 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.666538 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df29aa6b-58f4-48b9-87b6-09b6b50260e6-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:18.666661 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.666572 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df29aa6b-58f4-48b9-87b6-09b6b50260e6-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:18.668205 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.668176 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df29aa6b-58f4-48b9-87b6-09b6b50260e6-provider-credentials" pod "df29aa6b-58f4-48b9-87b6-09b6b50260e6" (UID: "df29aa6b-58f4-48b9-87b6-09b6b50260e6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:18.668313 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.668209 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df29aa6b-58f4-48b9-87b6-09b6b50260e6-kube-api-access-mpp6c" pod "df29aa6b-58f4-48b9-87b6-09b6b50260e6" (UID: "df29aa6b-58f4-48b9-87b6-09b6b50260e6"). InnerVolumeSpecName "kube-api-access-mpp6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:18.767887 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.767850 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpp6c\" (UniqueName: \"kubernetes.io/projected/df29aa6b-58f4-48b9-87b6-09b6b50260e6-kube-api-access-mpp6c\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:18.767887 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.767884 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df29aa6b-58f4-48b9-87b6-09b6b50260e6-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:18.943330 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.943205 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md"] Nov 03 17:10:18.954898 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:18.954861 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6b65945b4c-6c6md"] Nov 03 17:10:19.361724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:19.361029 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" Nov 03 17:10:19.429140 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:19.429088 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-l2682 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 03 17:10:19.429339 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:19.429156 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" podUID="629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 03 17:10:20.429473 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:20.429439 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-l2682 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 03 17:10:20.430029 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:20.429500 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" podUID="629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 03 17:10:20.568797 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:20.568764 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df29aa6b-58f4-48b9-87b6-09b6b50260e6" path="/var/lib/kubelet/pods/df29aa6b-58f4-48b9-87b6-09b6b50260e6/volumes" Nov 03 17:10:20.635584 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:20.635547 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3a0e27/agc-cred-d025f4746e15: secret "agc-cred-d025f4746e15" not found Nov 03 17:10:20.635824 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:20.635754 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl: secret "agc-cred-d025f4746e15" not found Nov 03 17:10:20.635902 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:20.635839 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-provider-credentials podName:e386c866-ccc7-4395-8866-b4650f18eb5e nodeName:}" failed. No retries permitted until 2026-11-03 17:10:21.135815315 +0000 UTC m=+3364.724815281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" (UID: "e386c866-ccc7-4395-8866-b4650f18eb5e") : secret "agc-cred-d025f4746e15" not found Nov 03 17:10:21.154720 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:21.148881 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3a0e27/agc-cred-d025f4746e15: secret "agc-cred-d025f4746e15" not found Nov 03 17:10:21.154720 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:21.148917 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl: secret "agc-cred-d025f4746e15" not found Nov 03 17:10:21.154720 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:21.148993 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-provider-credentials podName:e386c866-ccc7-4395-8866-b4650f18eb5e nodeName:}" failed. No retries permitted until 2026-11-03 17:10:22.148968623 +0000 UTC m=+3365.737968575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" (UID: "e386c866-ccc7-4395-8866-b4650f18eb5e") : secret "agc-cred-d025f4746e15" not found Nov 03 17:10:21.429230 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:21.429131 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-l2682 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 03 17:10:21.429230 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:21.429191 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" podUID="629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 03 17:10:21.535834 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:21.535790 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-559b8977dd-97dg9"] Nov 03 17:10:22.178521 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:22.178487 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3a0e27/agc-cred-d025f4746e15: secret "agc-cred-d025f4746e15" not found Nov 03 17:10:22.178521 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:22.178519 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl: secret "agc-cred-d025f4746e15" not found Nov 03 17:10:22.178871 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:22.178573 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-provider-credentials podName:e386c866-ccc7-4395-8866-b4650f18eb5e nodeName:}" failed. No retries permitted until 2026-11-03 17:10:24.178559621 +0000 UTC m=+3367.767559569 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" (UID: "e386c866-ccc7-4395-8866-b4650f18eb5e") : secret "agc-cred-d025f4746e15" not found Nov 03 17:10:22.429650 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:22.429561 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-l2682 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 03 17:10:22.429901 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:22.429872 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" podUID="629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 03 17:10:23.434772 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:23.432173 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-l2682 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 03 17:10:23.434772 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:23.432243 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" podUID="629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 03 17:10:24.266181 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:24.266107 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3a0e27/agc-cred-d025f4746e15: secret "agc-cred-d025f4746e15" not found Nov 03 17:10:24.266181 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:24.266133 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl: secret "agc-cred-d025f4746e15" not found Nov 03 17:10:24.266181 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:24.266189 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-provider-credentials podName:e386c866-ccc7-4395-8866-b4650f18eb5e nodeName:}" failed. No retries permitted until 2026-11-03 17:10:28.266175897 +0000 UTC m=+3371.855175844 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" (UID: "e386c866-ccc7-4395-8866-b4650f18eb5e") : secret "agc-cred-d025f4746e15" not found Nov 03 17:10:24.282018 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.281976 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf"] Nov 03 17:10:24.282968 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.282943 4947 cpu_manager.go:414] "containerMap: removing container" podUID="df29aa6b-58f4-48b9-87b6-09b6b50260e6" containerName="payload-processing" Nov 03 17:10:24.282968 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.282964 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df29aa6b-58f4-48b9-87b6-09b6b50260e6" containerName="payload-processing" Nov 03 17:10:24.283238 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.283076 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="df29aa6b-58f4-48b9-87b6-09b6b50260e6" containerName="payload-processing" Nov 03 17:10:24.288277 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.288254 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" Nov 03 17:10:24.310324 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.310299 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf"] Nov 03 17:10:24.367535 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.367491 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8kt5\" (UniqueName: \"kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-kube-api-access-h8kt5\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf\" (UID: \"4930988e-0835-4e65-9223-c56fd05d5d61\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" Nov 03 17:10:24.367782 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.367752 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf\" (UID: \"4930988e-0835-4e65-9223-c56fd05d5d61\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" Nov 03 17:10:24.368055 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.368031 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf\" (UID: \"4930988e-0835-4e65-9223-c56fd05d5d61\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" Nov 03 17:10:24.368115 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.368083 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf\" (UID: \"4930988e-0835-4e65-9223-c56fd05d5d61\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" Nov 03 17:10:24.429841 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.429800 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-l2682 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 03 17:10:24.430036 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.429891 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" podUID="629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 03 17:10:24.469614 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.469579 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf\" (UID: \"4930988e-0835-4e65-9223-c56fd05d5d61\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" Nov 03 17:10:24.470520 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.469739 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf\" (UID: \"4930988e-0835-4e65-9223-c56fd05d5d61\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" Nov 03 17:10:24.470520 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.469768 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf\" (UID: \"4930988e-0835-4e65-9223-c56fd05d5d61\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" Nov 03 17:10:24.470520 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.469948 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8kt5\" (UniqueName: \"kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-kube-api-access-h8kt5\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf\" (UID: \"4930988e-0835-4e65-9223-c56fd05d5d61\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" Nov 03 17:10:24.470745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.470610 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf\" (UID: \"4930988e-0835-4e65-9223-c56fd05d5d61\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" Nov 03 17:10:24.471005 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.470978 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf\" (UID: \"4930988e-0835-4e65-9223-c56fd05d5d61\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" Nov 03 17:10:24.473048 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.473017 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf\" (UID: \"4930988e-0835-4e65-9223-c56fd05d5d61\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" Nov 03 17:10:24.495453 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.495421 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8kt5\" (UniqueName: \"kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-kube-api-access-h8kt5\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf\" (UID: \"4930988e-0835-4e65-9223-c56fd05d5d61\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" Nov 03 17:10:24.603745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.603648 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" Nov 03 17:10:24.930766 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:24.930731 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf"] Nov 03 17:10:25.439723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.433636 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-l2682 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Nov 03 17:10:25.439723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.433720 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" podUID="629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Nov 03 17:10:25.499656 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.499618 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-559b8977dd-pztc5"] Nov 03 17:10:25.504583 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.504556 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" Nov 03 17:10:25.524614 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.524580 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-wq28z\"" Nov 03 17:10:25.525910 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.525884 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-559b8977dd-pztc5"] Nov 03 17:10:25.597111 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.597073 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whs7f\" (UniqueName: \"kubernetes.io/projected/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-kube-api-access-whs7f\") pod \"payload-processing-external-model-559b8977dd-pztc5\" (UID: \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" Nov 03 17:10:25.597302 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.597271 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-plugins-config-volume\") pod \"payload-processing-external-model-559b8977dd-pztc5\" (UID: \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" Nov 03 17:10:25.597472 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.597454 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-provider-credentials\") pod \"payload-processing-external-model-559b8977dd-pztc5\" (UID: \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" Nov 03 17:10:25.597568 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.597553 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-routing-overlay-volume\") pod \"payload-processing-external-model-559b8977dd-pztc5\" (UID: \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" Nov 03 17:10:25.680995 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.680959 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" event={"ID":"4930988e-0835-4e65-9223-c56fd05d5d61","Type":"ContainerStarted","Data":"dde5bd52e56b2391595493e26c7c48c29c271152d687340d86004f713e4fdf84"} Nov 03 17:10:25.681156 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.681017 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" Nov 03 17:10:25.681156 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.681034 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" event={"ID":"4930988e-0835-4e65-9223-c56fd05d5d61","Type":"ContainerStarted","Data":"232e41ea8613ef631f317a2fb5f6549f5aa04cc6851e7208909f2f0454bb97d5"} Nov 03 17:10:25.698756 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.698636 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-provider-credentials\") pod \"payload-processing-external-model-559b8977dd-pztc5\" (UID: \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" Nov 03 17:10:25.698756 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.698715 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-routing-overlay-volume\") pod \"payload-processing-external-model-559b8977dd-pztc5\" (UID: \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" Nov 03 17:10:25.699017 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.698919 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whs7f\" (UniqueName: \"kubernetes.io/projected/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-kube-api-access-whs7f\") pod \"payload-processing-external-model-559b8977dd-pztc5\" (UID: \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" Nov 03 17:10:25.700187 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:25.699445 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1f83c0/agc-cred-0874b9379bdd: secret "agc-cred-0874b9379bdd" not found Nov 03 17:10:25.700187 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:25.699470 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1f83c0/agc-cred-cb769e926784: secret "agc-cred-cb769e926784" not found Nov 03 17:10:25.700187 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:25.699481 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8: [secret "agc-cred-0874b9379bdd" not found, secret "agc-cred-cb769e926784" not found] Nov 03 17:10:25.700187 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.699502 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-plugins-config-volume\") pod \"payload-processing-external-model-559b8977dd-pztc5\" (UID: \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" Nov 03 17:10:25.700187 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:25.699536 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-provider-credentials podName:80ffba80-112d-45c3-b127-40a468224b7b nodeName:}" failed. No retries permitted until 2026-11-03 17:10:26.199518856 +0000 UTC m=+3369.788518817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" (UID: "80ffba80-112d-45c3-b127-40a468224b7b") : [secret "agc-cred-0874b9379bdd" not found, secret "agc-cred-cb769e926784" not found] Nov 03 17:10:25.700187 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.699585 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-routing-overlay-volume\") pod \"payload-processing-external-model-559b8977dd-pztc5\" (UID: \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" Nov 03 17:10:25.700631 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.700394 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-plugins-config-volume\") pod \"payload-processing-external-model-559b8977dd-pztc5\" (UID: \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" Nov 03 17:10:25.702946 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.702920 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-provider-credentials\") pod \"payload-processing-external-model-559b8977dd-pztc5\" (UID: \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" Nov 03 17:10:25.725415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.725327 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whs7f\" (UniqueName: \"kubernetes.io/projected/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-kube-api-access-whs7f\") pod \"payload-processing-external-model-559b8977dd-pztc5\" (UID: \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\") " pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" Nov 03 17:10:25.734326 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.734266 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" podStartSLOduration=1.7342517979999998 podStartE2EDuration="1.734251798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:25.728108911 +0000 UTC m=+3369.317108881" watchObservedRunningTime="2026-11-03 17:10:25.734251798 +0000 UTC m=+3369.323251767" Nov 03 17:10:25.811720 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:25.808198 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1f83c0/agc-cred-0874b9379bdd: secret "agc-cred-0874b9379bdd" not found Nov 03 17:10:25.811720 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:25.808229 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf: secret "agc-cred-0874b9379bdd" not found Nov 03 17:10:25.811720 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:25.808303 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-provider-credentials podName:4930988e-0835-4e65-9223-c56fd05d5d61 nodeName:}" failed. No retries permitted until 2026-11-03 17:10:26.308280329 +0000 UTC m=+3369.897280279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" (UID: "4930988e-0835-4e65-9223-c56fd05d5d61") : secret "agc-cred-0874b9379bdd" not found Nov 03 17:10:25.825722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:25.824285 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" Nov 03 17:10:26.046006 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:26.045082 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8"] Nov 03 17:10:26.053664 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:26.053582 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf"] Nov 03 17:10:26.247971 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:26.247940 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1f83c0/agc-cred-0874b9379bdd: secret "agc-cred-0874b9379bdd" not found Nov 03 17:10:26.247971 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:26.247967 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1f83c0/agc-cred-cb769e926784: secret "agc-cred-cb769e926784" not found Nov 03 17:10:26.247971 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:26.247976 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8: [secret "agc-cred-0874b9379bdd" not found, secret "agc-cred-cb769e926784" not found] Nov 03 17:10:26.248245 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:26.248031 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-provider-credentials podName:80ffba80-112d-45c3-b127-40a468224b7b nodeName:}" failed. No retries permitted until 2026-11-03 17:10:27.248017759 +0000 UTC m=+3370.837017709 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" (UID: "80ffba80-112d-45c3-b127-40a468224b7b") : [secret "agc-cred-0874b9379bdd" not found, secret "agc-cred-cb769e926784" not found] Nov 03 17:10:26.350085 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:26.350058 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1f83c0/agc-cred-0874b9379bdd: secret "agc-cred-0874b9379bdd" not found Nov 03 17:10:26.350085 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:26.350085 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf: secret "agc-cred-0874b9379bdd" not found Nov 03 17:10:26.350457 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:26.350154 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-provider-credentials podName:4930988e-0835-4e65-9223-c56fd05d5d61 nodeName:}" failed. No retries permitted until 2026-11-03 17:10:27.350134931 +0000 UTC m=+3370.939134900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" (UID: "4930988e-0835-4e65-9223-c56fd05d5d61") : secret "agc-cred-0874b9379bdd" not found Nov 03 17:10:26.352018 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:10:26.351993 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c4e856_1c20_4229_9e2f_0878fa65ad8e.slice/crio-2cc1d0b3072e772e949a728c433aa0ff73492b4f7d634f68e878fb8ee5f56bc0 WatchSource:0}: Error finding container 2cc1d0b3072e772e949a728c433aa0ff73492b4f7d634f68e878fb8ee5f56bc0: Status 404 returned error can't find the container with id 2cc1d0b3072e772e949a728c433aa0ff73492b4f7d634f68e878fb8ee5f56bc0 Nov 03 17:10:26.352955 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:26.352863 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-559b8977dd-pztc5"] Nov 03 17:10:26.542888 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:26.542833 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" podUID="4332898c-f56e-4ab6-b099-988456a34bf7" containerName="payload-processing" containerID="cri-o://43fb203fb5f89fc78b2717af235d9a415f2ae4eb05ecdbbc37fed5a8539ce1b2" gracePeriod=25 Nov 03 17:10:26.693361 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:26.693317 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" event={"ID":"a5c4e856-1c20-4229-9e2f-0878fa65ad8e","Type":"ContainerStarted","Data":"2cc1d0b3072e772e949a728c433aa0ff73492b4f7d634f68e878fb8ee5f56bc0"} Nov 03 17:10:26.693684 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:26.693661 4947 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" secret="" err="secret \"payload-processing-external-model-e2e--91a4f230-dockercfg-dm6c8\" not found" Nov 03 17:10:26.830828 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:26.830796 4947 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-plugins-e2e-extmod-a-1f83c0: configmap "payload-processing-external-model-plugins-e2e-extmod-a-1f83c0" not found Nov 03 17:10:26.831050 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:26.830869 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-plugins-config-volume podName:4930988e-0835-4e65-9223-c56fd05d5d61 nodeName:}" failed. No retries permitted until 2026-11-03 17:10:27.33085059 +0000 UTC m=+3370.919850555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" (UID: "4930988e-0835-4e65-9223-c56fd05d5d61") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-1f83c0" not found Nov 03 17:10:26.901032 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:26.900988 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" Nov 03 17:10:27.033738 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.033673 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4332898c-f56e-4ab6-b099-988456a34bf7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4332898c-f56e-4ab6-b099-988456a34bf7-provider-credentials\") pod \"4332898c-f56e-4ab6-b099-988456a34bf7\" (UID: \"4332898c-f56e-4ab6-b099-988456a34bf7\") " Nov 03 17:10:27.033911 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.033756 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4332898c-f56e-4ab6-b099-988456a34bf7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4332898c-f56e-4ab6-b099-988456a34bf7-plugins-config-volume\") pod \"4332898c-f56e-4ab6-b099-988456a34bf7\" (UID: \"4332898c-f56e-4ab6-b099-988456a34bf7\") " Nov 03 17:10:27.033952 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.033909 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4332898c-f56e-4ab6-b099-988456a34bf7-kube-api-access-gxxzv\" (UniqueName: \"kubernetes.io/projected/4332898c-f56e-4ab6-b099-988456a34bf7-kube-api-access-gxxzv\") pod \"4332898c-f56e-4ab6-b099-988456a34bf7\" (UID: \"4332898c-f56e-4ab6-b099-988456a34bf7\") " Nov 03 17:10:27.034214 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.034095 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4332898c-f56e-4ab6-b099-988456a34bf7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4332898c-f56e-4ab6-b099-988456a34bf7-routing-overlay-volume\") pod \"4332898c-f56e-4ab6-b099-988456a34bf7\" (UID: \"4332898c-f56e-4ab6-b099-988456a34bf7\") " Nov 03 17:10:27.034889 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.034860 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4332898c-f56e-4ab6-b099-988456a34bf7-plugins-config-volume" pod "4332898c-f56e-4ab6-b099-988456a34bf7" (UID: "4332898c-f56e-4ab6-b099-988456a34bf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:27.035515 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.035201 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4332898c-f56e-4ab6-b099-988456a34bf7-routing-overlay-volume" pod "4332898c-f56e-4ab6-b099-988456a34bf7" (UID: "4332898c-f56e-4ab6-b099-988456a34bf7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:27.035893 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.035870 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4332898c-f56e-4ab6-b099-988456a34bf7-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:27.036045 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.036018 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4332898c-f56e-4ab6-b099-988456a34bf7-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:27.037975 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.037946 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4332898c-f56e-4ab6-b099-988456a34bf7-provider-credentials" pod "4332898c-f56e-4ab6-b099-988456a34bf7" (UID: "4332898c-f56e-4ab6-b099-988456a34bf7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:27.038337 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.038301 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4332898c-f56e-4ab6-b099-988456a34bf7-kube-api-access-gxxzv" pod "4332898c-f56e-4ab6-b099-988456a34bf7" (UID: "4332898c-f56e-4ab6-b099-988456a34bf7"). InnerVolumeSpecName "kube-api-access-gxxzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:27.137404 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.137318 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4332898c-f56e-4ab6-b099-988456a34bf7-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:27.137404 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.137350 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxxzv\" (UniqueName: \"kubernetes.io/projected/4332898c-f56e-4ab6-b099-988456a34bf7-kube-api-access-gxxzv\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:27.179891 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.179185 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:27.180769 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.180732 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:27.181533 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.181512 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:10:27.350665 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:27.350564 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1f83c0/agc-cred-0874b9379bdd: secret "agc-cred-0874b9379bdd" not found Nov 03 17:10:27.350665 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:27.350598 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf: secret "agc-cred-0874b9379bdd" not found Nov 03 17:10:27.350665 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:27.350658 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-provider-credentials podName:4930988e-0835-4e65-9223-c56fd05d5d61 nodeName:}" failed. No retries permitted until 2026-11-03 17:10:29.350638977 +0000 UTC m=+3372.939638947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" (UID: "4930988e-0835-4e65-9223-c56fd05d5d61") : secret "agc-cred-0874b9379bdd" not found Nov 03 17:10:27.351136 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:27.351054 4947 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-plugins-e2e-extmod-a-1f83c0: configmap "payload-processing-external-model-plugins-e2e-extmod-a-1f83c0" not found Nov 03 17:10:27.351136 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:27.351111 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-plugins-config-volume podName:4930988e-0835-4e65-9223-c56fd05d5d61 nodeName:}" failed. No retries permitted until 2026-11-03 17:10:28.351095124 +0000 UTC m=+3371.940095092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" (UID: "4930988e-0835-4e65-9223-c56fd05d5d61") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-1f83c0" not found Nov 03 17:10:27.351269 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:27.351184 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1f83c0/agc-cred-0874b9379bdd: secret "agc-cred-0874b9379bdd" not found Nov 03 17:10:27.351269 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:27.351197 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1f83c0/agc-cred-cb769e926784: secret "agc-cred-cb769e926784" not found Nov 03 17:10:27.351269 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:27.351209 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8: [secret "agc-cred-0874b9379bdd" not found, secret "agc-cred-cb769e926784" not found] Nov 03 17:10:27.351269 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:27.351245 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-provider-credentials podName:80ffba80-112d-45c3-b127-40a468224b7b nodeName:}" failed. No retries permitted until 2026-11-03 17:10:29.351233635 +0000 UTC m=+3372.940233583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" (UID: "80ffba80-112d-45c3-b127-40a468224b7b") : [secret "agc-cred-0874b9379bdd" not found, secret "agc-cred-cb769e926784" not found] Nov 03 17:10:27.703863 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.703825 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" event={"ID":"a5c4e856-1c20-4229-9e2f-0878fa65ad8e","Type":"ContainerStarted","Data":"515fda9aca66b3f40b348f93f211eed20f6fa52377173d13430d5176c5ccd6bb"} Nov 03 17:10:27.704311 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.703894 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" Nov 03 17:10:27.705383 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.705358 4947 generic.go:356] "Generic (PLEG): container finished" podID="4332898c-f56e-4ab6-b099-988456a34bf7" containerID="43fb203fb5f89fc78b2717af235d9a415f2ae4eb05ecdbbc37fed5a8539ce1b2" exitCode=0 Nov 03 17:10:27.705519 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.705409 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" Nov 03 17:10:27.705519 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.705437 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" event={"ID":"4332898c-f56e-4ab6-b099-988456a34bf7","Type":"ContainerDied","Data":"43fb203fb5f89fc78b2717af235d9a415f2ae4eb05ecdbbc37fed5a8539ce1b2"} Nov 03 17:10:27.705519 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.705488 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-559b8977dd-97dg9" event={"ID":"4332898c-f56e-4ab6-b099-988456a34bf7","Type":"ContainerDied","Data":"4f594d37c72adbf912ab4a8d7c336e0c2141285cce4bfbbb9d8968555f703035"} Nov 03 17:10:27.705687 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.705521 4947 scope.go:122] "RemoveContainer" containerID="43fb203fb5f89fc78b2717af235d9a415f2ae4eb05ecdbbc37fed5a8539ce1b2" Nov 03 17:10:27.718286 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.718270 4947 scope.go:122] "RemoveContainer" containerID="43fb203fb5f89fc78b2717af235d9a415f2ae4eb05ecdbbc37fed5a8539ce1b2" Nov 03 17:10:27.718593 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:27.718576 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43fb203fb5f89fc78b2717af235d9a415f2ae4eb05ecdbbc37fed5a8539ce1b2\": container with ID starting with 43fb203fb5f89fc78b2717af235d9a415f2ae4eb05ecdbbc37fed5a8539ce1b2 not found: ID does not exist" containerID="43fb203fb5f89fc78b2717af235d9a415f2ae4eb05ecdbbc37fed5a8539ce1b2" Nov 03 17:10:27.718639 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.718603 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43fb203fb5f89fc78b2717af235d9a415f2ae4eb05ecdbbc37fed5a8539ce1b2"} err="failed to get container status \"43fb203fb5f89fc78b2717af235d9a415f2ae4eb05ecdbbc37fed5a8539ce1b2\": rpc error: code = NotFound desc = could not find container \"43fb203fb5f89fc78b2717af235d9a415f2ae4eb05ecdbbc37fed5a8539ce1b2\": container with ID starting with 43fb203fb5f89fc78b2717af235d9a415f2ae4eb05ecdbbc37fed5a8539ce1b2 not found: ID does not exist" Nov 03 17:10:27.779592 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.779539 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" podStartSLOduration=2.779525607 podStartE2EDuration="2.779525607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:27.779365757 +0000 UTC m=+3371.368365725" watchObservedRunningTime="2026-11-03 17:10:27.779525607 +0000 UTC m=+3371.368525575" Nov 03 17:10:27.889964 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.889922 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-559b8977dd-97dg9"] Nov 03 17:10:27.913417 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:27.913372 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-559b8977dd-97dg9"] Nov 03 17:10:28.270192 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:28.270153 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3a0e27/agc-cred-d025f4746e15: secret "agc-cred-d025f4746e15" not found Nov 03 17:10:28.270192 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:28.270184 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl: secret "agc-cred-d025f4746e15" not found Nov 03 17:10:28.270475 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:28.270308 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-provider-credentials podName:e386c866-ccc7-4395-8866-b4650f18eb5e nodeName:}" failed. No retries permitted until 2026-11-03 17:10:36.270286109 +0000 UTC m=+3379.859286071 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" (UID: "e386c866-ccc7-4395-8866-b4650f18eb5e") : secret "agc-cred-d025f4746e15" not found Nov 03 17:10:28.372579 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:28.372544 4947 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-plugins-e2e-extmod-a-1f83c0: configmap "payload-processing-external-model-plugins-e2e-extmod-a-1f83c0" not found Nov 03 17:10:28.372858 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:28.372635 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-plugins-config-volume podName:4930988e-0835-4e65-9223-c56fd05d5d61 nodeName:}" failed. No retries permitted until 2026-11-03 17:10:30.372616627 +0000 UTC m=+3373.961616584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" (UID: "4930988e-0835-4e65-9223-c56fd05d5d61") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-1f83c0" not found Nov 03 17:10:28.571624 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:28.571534 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4332898c-f56e-4ab6-b099-988456a34bf7" path="/var/lib/kubelet/pods/4332898c-f56e-4ab6-b099-988456a34bf7/volumes" Nov 03 17:10:28.581863 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:28.581835 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" Nov 03 17:10:28.668722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:28.668351 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl"] Nov 03 17:10:28.681565 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:28.680966 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3a0e27/agc-cred-6cd2cc145091: secret "agc-cred-6cd2cc145091" not found Nov 03 17:10:28.681565 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:28.680991 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z: secret "agc-cred-6cd2cc145091" not found Nov 03 17:10:28.681565 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:28.681045 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-provider-credentials podName:609f8f4e-e735-4055-9712-ae87b32bbe51 nodeName:}" failed. No retries permitted until 2026-11-03 17:10:29.181027006 +0000 UTC m=+3372.770026953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" (UID: "609f8f4e-e735-4055-9712-ae87b32bbe51") : secret "agc-cred-6cd2cc145091" not found Nov 03 17:10:28.752373 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:28.752318 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z"] Nov 03 17:10:29.208758 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:29.208500 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3a0e27/agc-cred-6cd2cc145091: secret "agc-cred-6cd2cc145091" not found Nov 03 17:10:29.208758 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:29.208539 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z: secret "agc-cred-6cd2cc145091" not found Nov 03 17:10:29.208758 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:29.208609 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-provider-credentials podName:609f8f4e-e735-4055-9712-ae87b32bbe51 nodeName:}" failed. No retries permitted until 2026-11-03 17:10:30.208587246 +0000 UTC m=+3373.797587215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" (UID: "609f8f4e-e735-4055-9712-ae87b32bbe51") : secret "agc-cred-6cd2cc145091" not found Nov 03 17:10:29.420629 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:29.420593 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1f83c0/agc-cred-0874b9379bdd: secret "agc-cred-0874b9379bdd" not found Nov 03 17:10:29.420897 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:29.420881 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf: secret "agc-cred-0874b9379bdd" not found Nov 03 17:10:29.421094 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:29.421082 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-provider-credentials podName:4930988e-0835-4e65-9223-c56fd05d5d61 nodeName:}" failed. No retries permitted until 2026-11-03 17:10:33.421059691 +0000 UTC m=+3377.010059641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" (UID: "4930988e-0835-4e65-9223-c56fd05d5d61") : secret "agc-cred-0874b9379bdd" not found Nov 03 17:10:29.421616 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:29.421595 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1f83c0/agc-cred-0874b9379bdd: secret "agc-cred-0874b9379bdd" not found Nov 03 17:10:29.421779 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:29.421767 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-1f83c0/agc-cred-cb769e926784: secret "agc-cred-cb769e926784" not found Nov 03 17:10:29.421882 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:29.421872 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8: [secret "agc-cred-0874b9379bdd" not found, secret "agc-cred-cb769e926784" not found] Nov 03 17:10:29.422012 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:29.421999 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-provider-credentials podName:80ffba80-112d-45c3-b127-40a468224b7b nodeName:}" failed. No retries permitted until 2026-11-03 17:10:33.421980699 +0000 UTC m=+3377.010980654 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" (UID: "80ffba80-112d-45c3-b127-40a468224b7b") : [secret "agc-cred-0874b9379bdd" not found, secret "agc-cred-cb769e926784" not found] Nov 03 17:10:30.133022 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:30.132981 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:10:30.242585 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:30.242125 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3a0e27/agc-cred-6cd2cc145091: secret "agc-cred-6cd2cc145091" not found Nov 03 17:10:30.242585 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:30.242158 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z: secret "agc-cred-6cd2cc145091" not found Nov 03 17:10:30.242585 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:30.242226 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-provider-credentials podName:609f8f4e-e735-4055-9712-ae87b32bbe51 nodeName:}" failed. No retries permitted until 2026-11-03 17:10:32.242204854 +0000 UTC m=+3375.831204802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" (UID: "609f8f4e-e735-4055-9712-ae87b32bbe51") : secret "agc-cred-6cd2cc145091" not found Nov 03 17:10:30.445209 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:30.445127 4947 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-plugins-e2e-extmod-a-1f83c0: configmap "payload-processing-external-model-plugins-e2e-extmod-a-1f83c0" not found Nov 03 17:10:30.445369 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:30.445213 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-plugins-config-volume podName:4930988e-0835-4e65-9223-c56fd05d5d61 nodeName:}" failed. No retries permitted until 2026-11-03 17:10:34.445195192 +0000 UTC m=+3378.034195139 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" (UID: "4930988e-0835-4e65-9223-c56fd05d5d61") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-1f83c0" not found Nov 03 17:10:30.689239 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:30.689200 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-559b8977dd-pztc5"] Nov 03 17:10:30.927198 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:30.912768 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs"] Nov 03 17:10:30.927198 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:30.914087 4947 cpu_manager.go:414] "containerMap: removing container" podUID="4332898c-f56e-4ab6-b099-988456a34bf7" containerName="payload-processing" Nov 03 17:10:30.927198 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:30.914110 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4332898c-f56e-4ab6-b099-988456a34bf7" containerName="payload-processing" Nov 03 17:10:30.927198 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:30.914311 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="4332898c-f56e-4ab6-b099-988456a34bf7" containerName="payload-processing" Nov 03 17:10:31.060516 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.054958 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" podUID="80ffba80-112d-45c3-b127-40a468224b7b" containerName="payload-processing" containerID="cri-o://a06d3f4be3dc16f1a28c3805a4f91d0573d41fcc1917e6d8095641bcbb798bbc" gracePeriod=25 Nov 03 17:10:31.130350 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.130314 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs"] Nov 03 17:10:31.130508 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.130433 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.321762 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.308473 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.321762 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.308569 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.321762 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.308613 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.321762 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.308678 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqfmd\" (UniqueName: \"kubernetes.io/projected/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-kube-api-access-nqfmd\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.409721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.402982 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.409721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.403065 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.409721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.403205 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.409721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.403278 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.409721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.403490 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.504897 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.504847 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqfmd\" (UniqueName: \"kubernetes.io/projected/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-kube-api-access-nqfmd\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.505116 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.505007 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.505116 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.505052 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.505260 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.505158 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.505260 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.505212 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.505379 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.505308 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.505458 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.505443 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.505547 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.505529 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.505619 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.505579 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.505664 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.505625 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.506307 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.506276 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.506462 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.506333 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.506544 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.506519 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.507203 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.507173 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.515007 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.509411 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.515007 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.509642 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.549348 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.543165 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqfmd\" (UniqueName: \"kubernetes.io/projected/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-kube-api-access-nqfmd\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.563324 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.553281 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-r8jhs\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.645434 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.645405 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" Nov 03 17:10:31.656348 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.656316 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:31.709223 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.709135 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ffba80-112d-45c3-b127-40a468224b7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ffba80-112d-45c3-b127-40a468224b7b-plugins-config-volume\") pod \"80ffba80-112d-45c3-b127-40a468224b7b\" (UID: \"80ffba80-112d-45c3-b127-40a468224b7b\") " Nov 03 17:10:31.709405 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.709229 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-provider-credentials\") pod \"80ffba80-112d-45c3-b127-40a468224b7b\" (UID: \"80ffba80-112d-45c3-b127-40a468224b7b\") " Nov 03 17:10:31.709405 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.709251 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-kube-api-access-nzmdh\" (UniqueName: \"kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-kube-api-access-nzmdh\") pod \"80ffba80-112d-45c3-b127-40a468224b7b\" (UID: \"80ffba80-112d-45c3-b127-40a468224b7b\") " Nov 03 17:10:31.709405 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.709311 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ffba80-112d-45c3-b127-40a468224b7b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80ffba80-112d-45c3-b127-40a468224b7b-routing-overlay-volume\") pod \"80ffba80-112d-45c3-b127-40a468224b7b\" (UID: \"80ffba80-112d-45c3-b127-40a468224b7b\") " Nov 03 17:10:31.710246 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.710216 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ffba80-112d-45c3-b127-40a468224b7b-routing-overlay-volume" pod "80ffba80-112d-45c3-b127-40a468224b7b" (UID: "80ffba80-112d-45c3-b127-40a468224b7b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:31.710405 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.710286 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ffba80-112d-45c3-b127-40a468224b7b-plugins-config-volume" pod "80ffba80-112d-45c3-b127-40a468224b7b" (UID: "80ffba80-112d-45c3-b127-40a468224b7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:31.715081 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.715052 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-kube-api-access-nzmdh" pod "80ffba80-112d-45c3-b127-40a468224b7b" (UID: "80ffba80-112d-45c3-b127-40a468224b7b"). InnerVolumeSpecName "kube-api-access-nzmdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:31.727999 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.727954 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-provider-credentials" pod "80ffba80-112d-45c3-b127-40a468224b7b" (UID: "80ffba80-112d-45c3-b127-40a468224b7b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:31.774788 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.774743 4947 generic.go:356] "Generic (PLEG): container finished" podID="80ffba80-112d-45c3-b127-40a468224b7b" containerID="a06d3f4be3dc16f1a28c3805a4f91d0573d41fcc1917e6d8095641bcbb798bbc" exitCode=0 Nov 03 17:10:31.775001 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.774901 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" event={"ID":"80ffba80-112d-45c3-b127-40a468224b7b","Type":"ContainerDied","Data":"a06d3f4be3dc16f1a28c3805a4f91d0573d41fcc1917e6d8095641bcbb798bbc"} Nov 03 17:10:31.775001 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.774955 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" event={"ID":"80ffba80-112d-45c3-b127-40a468224b7b","Type":"ContainerDied","Data":"6b17e6b4d22e7883d5f571ff12bc5e8ae5a589505ebe39ae5ea59c25374bbf78"} Nov 03 17:10:31.775001 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.774986 4947 scope.go:122] "RemoveContainer" containerID="a06d3f4be3dc16f1a28c3805a4f91d0573d41fcc1917e6d8095641bcbb798bbc" Nov 03 17:10:31.775160 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.775115 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8" Nov 03 17:10:31.804104 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.804080 4947 scope.go:122] "RemoveContainer" containerID="a06d3f4be3dc16f1a28c3805a4f91d0573d41fcc1917e6d8095641bcbb798bbc" Nov 03 17:10:31.804505 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:31.804479 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a06d3f4be3dc16f1a28c3805a4f91d0573d41fcc1917e6d8095641bcbb798bbc\": container with ID starting with a06d3f4be3dc16f1a28c3805a4f91d0573d41fcc1917e6d8095641bcbb798bbc not found: ID does not exist" containerID="a06d3f4be3dc16f1a28c3805a4f91d0573d41fcc1917e6d8095641bcbb798bbc" Nov 03 17:10:31.804593 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.804518 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a06d3f4be3dc16f1a28c3805a4f91d0573d41fcc1917e6d8095641bcbb798bbc"} err="failed to get container status \"a06d3f4be3dc16f1a28c3805a4f91d0573d41fcc1917e6d8095641bcbb798bbc\": rpc error: code = NotFound desc = could not find container \"a06d3f4be3dc16f1a28c3805a4f91d0573d41fcc1917e6d8095641bcbb798bbc\": container with ID starting with a06d3f4be3dc16f1a28c3805a4f91d0573d41fcc1917e6d8095641bcbb798bbc not found: ID does not exist" Nov 03 17:10:31.814489 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.812478 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/80ffba80-112d-45c3-b127-40a468224b7b-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:31.814489 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.812509 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80ffba80-112d-45c3-b127-40a468224b7b-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:31.814489 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.812525 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:31.814489 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.812537 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzmdh\" (UniqueName: \"kubernetes.io/projected/80ffba80-112d-45c3-b127-40a468224b7b-kube-api-access-nzmdh\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:31.842575 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.842537 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8"] Nov 03 17:10:31.847155 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.847122 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-7d5dz76x8"] Nov 03 17:10:31.988637 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:10:31.988607 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfcceda1_486c_48fe_a25e_d7d37f97e5a3.slice/crio-2506d29dcd12280f2f791b6719402e5d66d65700e4b27203373f49b7b1ef4988 WatchSource:0}: Error finding container 2506d29dcd12280f2f791b6719402e5d66d65700e4b27203373f49b7b1ef4988: Status 404 returned error can't find the container with id 2506d29dcd12280f2f791b6719402e5d66d65700e4b27203373f49b7b1ef4988 Nov 03 17:10:31.990221 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.990192 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs"] Nov 03 17:10:31.991753 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.991716 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:10:31.991833 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.991794 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:10:31.991833 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:31.991824 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:10:32.325722 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:32.324764 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3a0e27/agc-cred-6cd2cc145091: secret "agc-cred-6cd2cc145091" not found Nov 03 17:10:32.325722 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:32.324799 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z: secret "agc-cred-6cd2cc145091" not found Nov 03 17:10:32.325722 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:32.324918 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-provider-credentials podName:609f8f4e-e735-4055-9712-ae87b32bbe51 nodeName:}" failed. No retries permitted until 2026-11-03 17:10:36.324864487 +0000 UTC m=+3379.913864458 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" (UID: "609f8f4e-e735-4055-9712-ae87b32bbe51") : secret "agc-cred-6cd2cc145091" not found Nov 03 17:10:32.596210 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:32.596125 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80ffba80-112d-45c3-b127-40a468224b7b" path="/var/lib/kubelet/pods/80ffba80-112d-45c3-b127-40a468224b7b/volumes" Nov 03 17:10:32.706492 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:32.706441 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" podUID="4930988e-0835-4e65-9223-c56fd05d5d61" containerName="payload-processing" containerID="cri-o://dde5bd52e56b2391595493e26c7c48c29c271152d687340d86004f713e4fdf84" gracePeriod=25 Nov 03 17:10:32.794988 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:32.794938 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" event={"ID":"cfcceda1-486c-48fe-a25e-d7d37f97e5a3","Type":"ContainerStarted","Data":"d6e8d70153e827d4e5eeb57bf67a7272f2a0ef232bcb615d65c6c5e37c9ac3d8"} Nov 03 17:10:32.795168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:32.795008 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" event={"ID":"cfcceda1-486c-48fe-a25e-d7d37f97e5a3","Type":"ContainerStarted","Data":"2506d29dcd12280f2f791b6719402e5d66d65700e4b27203373f49b7b1ef4988"} Nov 03 17:10:32.870445 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:32.870328 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" podStartSLOduration=2.870314355 podStartE2EDuration="2.870314355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:32.856611697 +0000 UTC m=+3376.445611665" watchObservedRunningTime="2026-11-03 17:10:32.870314355 +0000 UTC m=+3376.459314323" Nov 03 17:10:33.131216 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.131127 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:10:33.136237 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.136216 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" Nov 03 17:10:33.259115 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.259081 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-routing-overlay-volume\") pod \"4930988e-0835-4e65-9223-c56fd05d5d61\" (UID: \"4930988e-0835-4e65-9223-c56fd05d5d61\") " Nov 03 17:10:33.259328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.259220 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-plugins-config-volume\") pod \"4930988e-0835-4e65-9223-c56fd05d5d61\" (UID: \"4930988e-0835-4e65-9223-c56fd05d5d61\") " Nov 03 17:10:33.259328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.259281 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-kube-api-access-h8kt5\" (UniqueName: \"kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-kube-api-access-h8kt5\") pod \"4930988e-0835-4e65-9223-c56fd05d5d61\" (UID: \"4930988e-0835-4e65-9223-c56fd05d5d61\") " Nov 03 17:10:33.259328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.259322 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-provider-credentials\") pod \"4930988e-0835-4e65-9223-c56fd05d5d61\" (UID: \"4930988e-0835-4e65-9223-c56fd05d5d61\") " Nov 03 17:10:33.259787 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.259761 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-plugins-config-volume" pod "4930988e-0835-4e65-9223-c56fd05d5d61" (UID: "4930988e-0835-4e65-9223-c56fd05d5d61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:33.260238 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.260217 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-routing-overlay-volume" pod "4930988e-0835-4e65-9223-c56fd05d5d61" (UID: "4930988e-0835-4e65-9223-c56fd05d5d61"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:33.260650 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.260627 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:33.260831 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.260656 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4930988e-0835-4e65-9223-c56fd05d5d61-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:33.262323 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.262289 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-kube-api-access-h8kt5" pod "4930988e-0835-4e65-9223-c56fd05d5d61" (UID: "4930988e-0835-4e65-9223-c56fd05d5d61"). InnerVolumeSpecName "kube-api-access-h8kt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:33.262843 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.262807 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-provider-credentials" pod "4930988e-0835-4e65-9223-c56fd05d5d61" (UID: "4930988e-0835-4e65-9223-c56fd05d5d61"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:33.372746 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.371861 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8kt5\" (UniqueName: \"kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-kube-api-access-h8kt5\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:33.372746 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.371901 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4930988e-0835-4e65-9223-c56fd05d5d61-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:33.555783 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.555745 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb"] Nov 03 17:10:33.562021 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.561981 4947 cpu_manager.go:414] "containerMap: removing container" podUID="4930988e-0835-4e65-9223-c56fd05d5d61" containerName="payload-processing" Nov 03 17:10:33.562021 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.562019 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4930988e-0835-4e65-9223-c56fd05d5d61" containerName="payload-processing" Nov 03 17:10:33.562297 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.562079 4947 cpu_manager.go:414] "containerMap: removing container" podUID="80ffba80-112d-45c3-b127-40a468224b7b" containerName="payload-processing" Nov 03 17:10:33.562297 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.562087 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ffba80-112d-45c3-b127-40a468224b7b" containerName="payload-processing" Nov 03 17:10:33.562418 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.562333 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ffba80-112d-45c3-b127-40a468224b7b" containerName="payload-processing" Nov 03 17:10:33.562418 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.562355 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="4930988e-0835-4e65-9223-c56fd05d5d61" containerName="payload-processing" Nov 03 17:10:33.567103 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.567076 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb"] Nov 03 17:10:33.567415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.567394 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" Nov 03 17:10:33.583985 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.583953 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"payload-processing-external-model-plugins-e2e-extmod-a-2492c4\"" Nov 03 17:10:33.584159 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.584034 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"openshift-service-ca.crt\"" Nov 03 17:10:33.584159 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.584090 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"kube-root-ca.crt\"" Nov 03 17:10:33.584159 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.584128 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"routing-overlay\"" Nov 03 17:10:33.635048 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.632989 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"payload-processing-external-model-e2e--6d96ad03-dockercfg-68l2b\"" Nov 03 17:10:33.635048 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.633099 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-2492c4\"/\"agc-cred-a3d123b1c07d\"" Nov 03 17:10:33.657078 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.657033 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:33.658612 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.658584 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 03 17:10:33.658772 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.658633 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" podUID="cfcceda1-486c-48fe-a25e-d7d37f97e5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 03 17:10:33.669020 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.668973 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" podUID="e386c866-ccc7-4395-8866-b4650f18eb5e" containerName="payload-processing" containerID="cri-o://c1464655ccef48e4a2ca974d7a969dbfecad7224ba8d01bdbcb62014b4ef07d2" gracePeriod=25 Nov 03 17:10:33.684200 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.684166 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb\" (UID: \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" Nov 03 17:10:33.684505 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.684476 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb\" (UID: \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" Nov 03 17:10:33.684591 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.684544 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb\" (UID: \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" Nov 03 17:10:33.684865 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.684841 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97zqn\" (UniqueName: \"kubernetes.io/projected/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-kube-api-access-97zqn\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb\" (UID: \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" Nov 03 17:10:33.753577 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.753528 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" podUID="609f8f4e-e735-4055-9712-ae87b32bbe51" containerName="payload-processing" containerID="cri-o://0f99295e477a52e43d6636c99d8fb481436b0c0db897e90df33ec2a503a97b90" gracePeriod=25 Nov 03 17:10:33.800725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.794972 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97zqn\" (UniqueName: \"kubernetes.io/projected/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-kube-api-access-97zqn\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb\" (UID: \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" Nov 03 17:10:33.800725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.795194 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb\" (UID: \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" Nov 03 17:10:33.800725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.795328 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb\" (UID: \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" Nov 03 17:10:33.800725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.795374 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb\" (UID: \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" Nov 03 17:10:33.800725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.796818 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb\" (UID: \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" Nov 03 17:10:33.830816 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.828105 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb\" (UID: \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" Nov 03 17:10:33.867187 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.859444 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb\" (UID: \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" Nov 03 17:10:33.917722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.897053 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97zqn\" (UniqueName: \"kubernetes.io/projected/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-kube-api-access-97zqn\") pod \"payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb\" (UID: \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\") " pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" Nov 03 17:10:33.917722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.900375 4947 generic.go:356] "Generic (PLEG): container finished" podID="4930988e-0835-4e65-9223-c56fd05d5d61" containerID="dde5bd52e56b2391595493e26c7c48c29c271152d687340d86004f713e4fdf84" exitCode=0 Nov 03 17:10:33.917722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.901365 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" Nov 03 17:10:33.917722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.901772 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" event={"ID":"4930988e-0835-4e65-9223-c56fd05d5d61","Type":"ContainerDied","Data":"dde5bd52e56b2391595493e26c7c48c29c271152d687340d86004f713e4fdf84"} Nov 03 17:10:33.917722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.902454 4947 scope.go:122] "RemoveContainer" containerID="dde5bd52e56b2391595493e26c7c48c29c271152d687340d86004f713e4fdf84" Nov 03 17:10:33.917722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:33.902621 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf" event={"ID":"4930988e-0835-4e65-9223-c56fd05d5d61","Type":"ContainerDied","Data":"232e41ea8613ef631f317a2fb5f6549f5aa04cc6851e7208909f2f0454bb97d5"} Nov 03 17:10:34.079158 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.079110 4947 scope.go:122] "RemoveContainer" containerID="dde5bd52e56b2391595493e26c7c48c29c271152d687340d86004f713e4fdf84" Nov 03 17:10:34.080050 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:34.080013 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde5bd52e56b2391595493e26c7c48c29c271152d687340d86004f713e4fdf84\": container with ID starting with dde5bd52e56b2391595493e26c7c48c29c271152d687340d86004f713e4fdf84 not found: ID does not exist" containerID="dde5bd52e56b2391595493e26c7c48c29c271152d687340d86004f713e4fdf84" Nov 03 17:10:34.082169 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.080345 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde5bd52e56b2391595493e26c7c48c29c271152d687340d86004f713e4fdf84"} err="failed to get container status \"dde5bd52e56b2391595493e26c7c48c29c271152d687340d86004f713e4fdf84\": rpc error: code = NotFound desc = could not find container \"dde5bd52e56b2391595493e26c7c48c29c271152d687340d86004f713e4fdf84\": container with ID starting with dde5bd52e56b2391595493e26c7c48c29c271152d687340d86004f713e4fdf84 not found: ID does not exist" Nov 03 17:10:34.150579 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.150544 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf"] Nov 03 17:10:34.156307 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.156270 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-6b4d6gmdf"] Nov 03 17:10:34.184681 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.184657 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" Nov 03 17:10:34.256027 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.255995 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" Nov 03 17:10:34.374744 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.374522 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e386c866-ccc7-4395-8866-b4650f18eb5e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e386c866-ccc7-4395-8866-b4650f18eb5e-routing-overlay-volume\") pod \"e386c866-ccc7-4395-8866-b4650f18eb5e\" (UID: \"e386c866-ccc7-4395-8866-b4650f18eb5e\") " Nov 03 17:10:34.374744 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.374654 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-provider-credentials\") pod \"e386c866-ccc7-4395-8866-b4650f18eb5e\" (UID: \"e386c866-ccc7-4395-8866-b4650f18eb5e\") " Nov 03 17:10:34.376149 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.375334 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e386c866-ccc7-4395-8866-b4650f18eb5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e386c866-ccc7-4395-8866-b4650f18eb5e-plugins-config-volume\") pod \"e386c866-ccc7-4395-8866-b4650f18eb5e\" (UID: \"e386c866-ccc7-4395-8866-b4650f18eb5e\") " Nov 03 17:10:34.376149 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.375573 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-kube-api-access-69jhg\" (UniqueName: \"kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-kube-api-access-69jhg\") pod \"e386c866-ccc7-4395-8866-b4650f18eb5e\" (UID: \"e386c866-ccc7-4395-8866-b4650f18eb5e\") " Nov 03 17:10:34.378104 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.378079 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e386c866-ccc7-4395-8866-b4650f18eb5e-plugins-config-volume" pod "e386c866-ccc7-4395-8866-b4650f18eb5e" (UID: "e386c866-ccc7-4395-8866-b4650f18eb5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:34.378430 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.378410 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e386c866-ccc7-4395-8866-b4650f18eb5e-routing-overlay-volume" pod "e386c866-ccc7-4395-8866-b4650f18eb5e" (UID: "e386c866-ccc7-4395-8866-b4650f18eb5e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:34.380809 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.380783 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-kube-api-access-69jhg" pod "e386c866-ccc7-4395-8866-b4650f18eb5e" (UID: "e386c866-ccc7-4395-8866-b4650f18eb5e"). InnerVolumeSpecName "kube-api-access-69jhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:34.381676 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.381650 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-provider-credentials" pod "e386c866-ccc7-4395-8866-b4650f18eb5e" (UID: "e386c866-ccc7-4395-8866-b4650f18eb5e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:34.644305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.644207 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e386c866-ccc7-4395-8866-b4650f18eb5e-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:34.644305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.644248 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:34.644305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.644264 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e386c866-ccc7-4395-8866-b4650f18eb5e-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:34.644305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.644279 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69jhg\" (UniqueName: \"kubernetes.io/projected/e386c866-ccc7-4395-8866-b4650f18eb5e-kube-api-access-69jhg\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:34.662229 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.657668 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 03 17:10:34.662229 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.657752 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" podUID="cfcceda1-486c-48fe-a25e-d7d37f97e5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 03 17:10:34.683951 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.672935 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" Nov 03 17:10:34.757745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.746347 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/609f8f4e-e735-4055-9712-ae87b32bbe51-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/609f8f4e-e735-4055-9712-ae87b32bbe51-routing-overlay-volume\") pod \"609f8f4e-e735-4055-9712-ae87b32bbe51\" (UID: \"609f8f4e-e735-4055-9712-ae87b32bbe51\") " Nov 03 17:10:34.757745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.746496 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/609f8f4e-e735-4055-9712-ae87b32bbe51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609f8f4e-e735-4055-9712-ae87b32bbe51-plugins-config-volume\") pod \"609f8f4e-e735-4055-9712-ae87b32bbe51\" (UID: \"609f8f4e-e735-4055-9712-ae87b32bbe51\") " Nov 03 17:10:34.757745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.751739 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609f8f4e-e735-4055-9712-ae87b32bbe51-routing-overlay-volume" pod "609f8f4e-e735-4055-9712-ae87b32bbe51" (UID: "609f8f4e-e735-4055-9712-ae87b32bbe51"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:34.772550 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.770872 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-kube-api-access-g8wmc\" (UniqueName: \"kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-kube-api-access-g8wmc\") pod \"609f8f4e-e735-4055-9712-ae87b32bbe51\" (UID: \"609f8f4e-e735-4055-9712-ae87b32bbe51\") " Nov 03 17:10:34.772550 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.771174 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-provider-credentials\" (UniqueName: \"kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-provider-credentials\") pod \"609f8f4e-e735-4055-9712-ae87b32bbe51\" (UID: \"609f8f4e-e735-4055-9712-ae87b32bbe51\") " Nov 03 17:10:34.772550 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.772094 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/609f8f4e-e735-4055-9712-ae87b32bbe51-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:34.774780 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.773593 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4930988e-0835-4e65-9223-c56fd05d5d61" path="/var/lib/kubelet/pods/4930988e-0835-4e65-9223-c56fd05d5d61/volumes" Nov 03 17:10:34.775321 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.775160 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609f8f4e-e735-4055-9712-ae87b32bbe51-plugins-config-volume" pod "609f8f4e-e735-4055-9712-ae87b32bbe51" (UID: "609f8f4e-e735-4055-9712-ae87b32bbe51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:34.807744 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.804965 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb"] Nov 03 17:10:34.864354 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.864303 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-kube-api-access-g8wmc" pod "609f8f4e-e735-4055-9712-ae87b32bbe51" (UID: "609f8f4e-e735-4055-9712-ae87b32bbe51"). InnerVolumeSpecName "kube-api-access-g8wmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:34.865755 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.865707 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-provider-credentials" pod "609f8f4e-e735-4055-9712-ae87b32bbe51" (UID: "609f8f4e-e735-4055-9712-ae87b32bbe51"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:34.874012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.873984 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:34.874012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.874014 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609f8f4e-e735-4055-9712-ae87b32bbe51-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:34.874187 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:34.874023 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8wmc\" (UniqueName: \"kubernetes.io/projected/609f8f4e-e735-4055-9712-ae87b32bbe51-kube-api-access-g8wmc\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:35.194719 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.193246 4947 generic.go:356] "Generic (PLEG): container finished" podID="e386c866-ccc7-4395-8866-b4650f18eb5e" containerID="c1464655ccef48e4a2ca974d7a969dbfecad7224ba8d01bdbcb62014b4ef07d2" exitCode=0 Nov 03 17:10:35.194719 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.193552 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" event={"ID":"e386c866-ccc7-4395-8866-b4650f18eb5e","Type":"ContainerDied","Data":"c1464655ccef48e4a2ca974d7a969dbfecad7224ba8d01bdbcb62014b4ef07d2"} Nov 03 17:10:35.194719 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.193625 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" event={"ID":"e386c866-ccc7-4395-8866-b4650f18eb5e","Type":"ContainerDied","Data":"633810d179ca05c62c1da7e86bb571c9a0572c0b8209a363a9429e54473271f4"} Nov 03 17:10:35.194719 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.193660 4947 scope.go:122] "RemoveContainer" containerID="c1464655ccef48e4a2ca974d7a969dbfecad7224ba8d01bdbcb62014b4ef07d2" Nov 03 17:10:35.194719 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.193867 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl" Nov 03 17:10:35.237721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.215407 4947 generic.go:356] "Generic (PLEG): container finished" podID="609f8f4e-e735-4055-9712-ae87b32bbe51" containerID="0f99295e477a52e43d6636c99d8fb481436b0c0db897e90df33ec2a503a97b90" exitCode=0 Nov 03 17:10:35.237721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.215563 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" event={"ID":"609f8f4e-e735-4055-9712-ae87b32bbe51","Type":"ContainerDied","Data":"0f99295e477a52e43d6636c99d8fb481436b0c0db897e90df33ec2a503a97b90"} Nov 03 17:10:35.237721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.215616 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" event={"ID":"609f8f4e-e735-4055-9712-ae87b32bbe51","Type":"ContainerDied","Data":"2333c99007cdd0bce64333680f5ec97c3a0db90c9fcc1e73f50946da8221fa9b"} Nov 03 17:10:35.237721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.215724 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z" Nov 03 17:10:35.408959 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.406781 4947 scope.go:122] "RemoveContainer" containerID="c1464655ccef48e4a2ca974d7a969dbfecad7224ba8d01bdbcb62014b4ef07d2" Nov 03 17:10:35.433723 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:35.420056 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1464655ccef48e4a2ca974d7a969dbfecad7224ba8d01bdbcb62014b4ef07d2\": container with ID starting with c1464655ccef48e4a2ca974d7a969dbfecad7224ba8d01bdbcb62014b4ef07d2 not found: ID does not exist" containerID="c1464655ccef48e4a2ca974d7a969dbfecad7224ba8d01bdbcb62014b4ef07d2" Nov 03 17:10:35.433723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.420101 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1464655ccef48e4a2ca974d7a969dbfecad7224ba8d01bdbcb62014b4ef07d2"} err="failed to get container status \"c1464655ccef48e4a2ca974d7a969dbfecad7224ba8d01bdbcb62014b4ef07d2\": rpc error: code = NotFound desc = could not find container \"c1464655ccef48e4a2ca974d7a969dbfecad7224ba8d01bdbcb62014b4ef07d2\": container with ID starting with c1464655ccef48e4a2ca974d7a969dbfecad7224ba8d01bdbcb62014b4ef07d2 not found: ID does not exist" Nov 03 17:10:35.433723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.420128 4947 scope.go:122] "RemoveContainer" containerID="0f99295e477a52e43d6636c99d8fb481436b0c0db897e90df33ec2a503a97b90" Nov 03 17:10:35.433723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.420243 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" event={"ID":"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62","Type":"ContainerStarted","Data":"e467c4962b50af9dddb1c176c66353091dacf5763260964b016acd452e34a6dc"} Nov 03 17:10:35.506513 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.505595 4947 scope.go:122] "RemoveContainer" containerID="0f99295e477a52e43d6636c99d8fb481436b0c0db897e90df33ec2a503a97b90" Nov 03 17:10:35.517721 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:35.515249 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f99295e477a52e43d6636c99d8fb481436b0c0db897e90df33ec2a503a97b90\": container with ID starting with 0f99295e477a52e43d6636c99d8fb481436b0c0db897e90df33ec2a503a97b90 not found: ID does not exist" containerID="0f99295e477a52e43d6636c99d8fb481436b0c0db897e90df33ec2a503a97b90" Nov 03 17:10:35.517721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.515300 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f99295e477a52e43d6636c99d8fb481436b0c0db897e90df33ec2a503a97b90"} err="failed to get container status \"0f99295e477a52e43d6636c99d8fb481436b0c0db897e90df33ec2a503a97b90\": rpc error: code = NotFound desc = could not find container \"0f99295e477a52e43d6636c99d8fb481436b0c0db897e90df33ec2a503a97b90\": container with ID starting with 0f99295e477a52e43d6636c99d8fb481436b0c0db897e90df33ec2a503a97b90 not found: ID does not exist" Nov 03 17:10:35.543733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.540288 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl"] Nov 03 17:10:35.598442 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.595043 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-7557vb9pl"] Nov 03 17:10:35.639728 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.638766 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z"] Nov 03 17:10:35.661725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.657245 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 03 17:10:35.661725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.657307 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" podUID="cfcceda1-486c-48fe-a25e-d7d37f97e5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 03 17:10:35.687722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.684419 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-9d75txk2z"] Nov 03 17:10:35.697901 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.697832 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" podUID="a5c4e856-1c20-4229-9e2f-0878fa65ad8e" containerName="payload-processing" containerID="cri-o://515fda9aca66b3f40b348f93f211eed20f6fa52377173d13430d5176c5ccd6bb" gracePeriod=25 Nov 03 17:10:35.919722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:35.918737 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:10:36.313775 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.311890 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" Nov 03 17:10:36.407417 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.407367 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-routing-overlay-volume\") pod \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\" (UID: \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\") " Nov 03 17:10:36.431984 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.431944 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-kube-api-access-whs7f\" (UniqueName: \"kubernetes.io/projected/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-kube-api-access-whs7f\") pod \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\" (UID: \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\") " Nov 03 17:10:36.432628 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.432603 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-provider-credentials\") pod \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\" (UID: \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\") " Nov 03 17:10:36.433041 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.433016 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-plugins-config-volume\") pod \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\" (UID: \"a5c4e856-1c20-4229-9e2f-0878fa65ad8e\") " Nov 03 17:10:36.434402 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.415279 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-routing-overlay-volume" pod "a5c4e856-1c20-4229-9e2f-0878fa65ad8e" (UID: "a5c4e856-1c20-4229-9e2f-0878fa65ad8e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:36.434532 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.434375 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-plugins-config-volume" pod "a5c4e856-1c20-4229-9e2f-0878fa65ad8e" (UID: "a5c4e856-1c20-4229-9e2f-0878fa65ad8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:10:36.436290 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.436262 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-kube-api-access-whs7f" pod "a5c4e856-1c20-4229-9e2f-0878fa65ad8e" (UID: "a5c4e856-1c20-4229-9e2f-0878fa65ad8e"). InnerVolumeSpecName "kube-api-access-whs7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:36.453501 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.453463 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-provider-credentials" pod "a5c4e856-1c20-4229-9e2f-0878fa65ad8e" (UID: "a5c4e856-1c20-4229-9e2f-0878fa65ad8e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:10:36.458586 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.458172 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" event={"ID":"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62","Type":"ContainerStarted","Data":"4da11dd4771a590082cfec52fb377cf50de1099afca5bf1ca9acc30687df599b"} Nov 03 17:10:36.462356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.461980 4947 generic.go:356] "Generic (PLEG): container finished" podID="a5c4e856-1c20-4229-9e2f-0878fa65ad8e" containerID="515fda9aca66b3f40b348f93f211eed20f6fa52377173d13430d5176c5ccd6bb" exitCode=0 Nov 03 17:10:36.462356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.462125 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" event={"ID":"a5c4e856-1c20-4229-9e2f-0878fa65ad8e","Type":"ContainerDied","Data":"515fda9aca66b3f40b348f93f211eed20f6fa52377173d13430d5176c5ccd6bb"} Nov 03 17:10:36.462356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.462177 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" event={"ID":"a5c4e856-1c20-4229-9e2f-0878fa65ad8e","Type":"ContainerDied","Data":"2cc1d0b3072e772e949a728c433aa0ff73492b4f7d634f68e878fb8ee5f56bc0"} Nov 03 17:10:36.462356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.462209 4947 scope.go:122] "RemoveContainer" containerID="515fda9aca66b3f40b348f93f211eed20f6fa52377173d13430d5176c5ccd6bb" Nov 03 17:10:36.465992 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.465940 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-559b8977dd-pztc5" Nov 03 17:10:36.466939 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.466780 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" Nov 03 17:10:36.491259 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.491226 4947 scope.go:122] "RemoveContainer" containerID="515fda9aca66b3f40b348f93f211eed20f6fa52377173d13430d5176c5ccd6bb" Nov 03 17:10:36.491687 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:10:36.491653 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"515fda9aca66b3f40b348f93f211eed20f6fa52377173d13430d5176c5ccd6bb\": container with ID starting with 515fda9aca66b3f40b348f93f211eed20f6fa52377173d13430d5176c5ccd6bb not found: ID does not exist" containerID="515fda9aca66b3f40b348f93f211eed20f6fa52377173d13430d5176c5ccd6bb" Nov 03 17:10:36.492094 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.492068 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"515fda9aca66b3f40b348f93f211eed20f6fa52377173d13430d5176c5ccd6bb"} err="failed to get container status \"515fda9aca66b3f40b348f93f211eed20f6fa52377173d13430d5176c5ccd6bb\": rpc error: code = NotFound desc = could not find container \"515fda9aca66b3f40b348f93f211eed20f6fa52377173d13430d5176c5ccd6bb\": container with ID starting with 515fda9aca66b3f40b348f93f211eed20f6fa52377173d13430d5176c5ccd6bb not found: ID does not exist" Nov 03 17:10:36.538725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.538657 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:36.539068 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.539048 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:36.539221 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.539207 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whs7f\" (UniqueName: \"kubernetes.io/projected/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-kube-api-access-whs7f\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:36.539318 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.539307 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a5c4e856-1c20-4229-9e2f-0878fa65ad8e-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:10:36.541078 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.540873 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" podStartSLOduration=3.540857115 podStartE2EDuration="3.540857115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:36.536231269 +0000 UTC m=+3380.125231239" watchObservedRunningTime="2026-11-03 17:10:36.540857115 +0000 UTC m=+3380.129857086" Nov 03 17:10:36.610965 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.610874 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="609f8f4e-e735-4055-9712-ae87b32bbe51" path="/var/lib/kubelet/pods/609f8f4e-e735-4055-9712-ae87b32bbe51/volumes" Nov 03 17:10:36.616160 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.616122 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e386c866-ccc7-4395-8866-b4650f18eb5e" path="/var/lib/kubelet/pods/e386c866-ccc7-4395-8866-b4650f18eb5e/volumes" Nov 03 17:10:36.616959 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.616931 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-559b8977dd-pztc5"] Nov 03 17:10:36.626725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.623292 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-559b8977dd-pztc5"] Nov 03 17:10:36.682735 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.679910 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 03 17:10:36.682735 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:36.679977 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" podUID="cfcceda1-486c-48fe-a25e-d7d37f97e5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 03 17:10:37.669846 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:37.669810 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 03 17:10:37.670457 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:37.670423 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" podUID="cfcceda1-486c-48fe-a25e-d7d37f97e5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 03 17:10:38.590661 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:38.590630 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5c4e856-1c20-4229-9e2f-0878fa65ad8e" path="/var/lib/kubelet/pods/a5c4e856-1c20-4229-9e2f-0878fa65ad8e/volumes" Nov 03 17:10:38.663727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:38.660074 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 03 17:10:38.663727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:38.660154 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" podUID="cfcceda1-486c-48fe-a25e-d7d37f97e5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 03 17:10:39.656896 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:39.656863 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 03 17:10:39.657337 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:39.656921 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" podUID="cfcceda1-486c-48fe-a25e-d7d37f97e5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 03 17:10:40.658503 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:40.658468 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Nov 03 17:10:40.659225 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:40.658525 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" podUID="cfcceda1-486c-48fe-a25e-d7d37f97e5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Nov 03 17:10:42.392650 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:42.392602 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:42.393382 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:42.393353 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:42.395095 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:42.395074 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:10:44.012646 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.012604 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5"] Nov 03 17:10:44.014031 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.013962 4947 cpu_manager.go:414] "containerMap: removing container" podUID="a5c4e856-1c20-4229-9e2f-0878fa65ad8e" containerName="payload-processing" Nov 03 17:10:44.014031 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.013988 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5c4e856-1c20-4229-9e2f-0878fa65ad8e" containerName="payload-processing" Nov 03 17:10:44.014313 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.014008 4947 cpu_manager.go:414] "containerMap: removing container" podUID="609f8f4e-e735-4055-9712-ae87b32bbe51" containerName="payload-processing" Nov 03 17:10:44.014313 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.014093 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="609f8f4e-e735-4055-9712-ae87b32bbe51" containerName="payload-processing" Nov 03 17:10:44.014313 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.014132 4947 cpu_manager.go:414] "containerMap: removing container" podUID="e386c866-ccc7-4395-8866-b4650f18eb5e" containerName="payload-processing" Nov 03 17:10:44.014313 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.014140 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e386c866-ccc7-4395-8866-b4650f18eb5e" containerName="payload-processing" Nov 03 17:10:44.014440 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.014393 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5c4e856-1c20-4229-9e2f-0878fa65ad8e" containerName="payload-processing" Nov 03 17:10:44.014440 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.014408 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="609f8f4e-e735-4055-9712-ae87b32bbe51" containerName="payload-processing" Nov 03 17:10:44.014440 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.014423 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e386c866-ccc7-4395-8866-b4650f18eb5e" containerName="payload-processing" Nov 03 17:10:44.020904 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.020878 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" Nov 03 17:10:44.036920 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.036142 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5"] Nov 03 17:10:44.041408 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.041381 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1f83c0\"/\"openshift-service-ca.crt\"" Nov 03 17:10:44.041580 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.041390 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1f83c0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-1f83c0\"" Nov 03 17:10:44.041660 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.041480 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1f83c0\"/\"kube-root-ca.crt\"" Nov 03 17:10:44.055266 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.054968 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-1f83c0\"/\"routing-overlay\"" Nov 03 17:10:44.057447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.057428 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1f83c0\"/\"agc-cred-756f3de6dedf\"" Nov 03 17:10:44.057647 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.057533 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1f83c0\"/\"payload-processing-external-model-e2e--91a4f230-dockercfg-cl7hq\"" Nov 03 17:10:44.074726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.074673 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5\" (UID: \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" Nov 03 17:10:44.074879 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.074838 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5\" (UID: \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" Nov 03 17:10:44.074970 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.074952 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5\" (UID: \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" Nov 03 17:10:44.075181 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.075150 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk6hh\" (UniqueName: \"kubernetes.io/projected/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-kube-api-access-qk6hh\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5\" (UID: \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" Nov 03 17:10:44.176408 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.176369 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5\" (UID: \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" Nov 03 17:10:44.176603 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.176496 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk6hh\" (UniqueName: \"kubernetes.io/projected/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-kube-api-access-qk6hh\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5\" (UID: \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" Nov 03 17:10:44.176677 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.176613 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5\" (UID: \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" Nov 03 17:10:44.176795 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.176779 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5\" (UID: \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" Nov 03 17:10:44.177285 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.177245 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5\" (UID: \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" Nov 03 17:10:44.177467 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.177444 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5\" (UID: \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" Nov 03 17:10:44.179396 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.179370 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5\" (UID: \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" Nov 03 17:10:44.237259 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.218436 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk6hh\" (UniqueName: \"kubernetes.io/projected/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-kube-api-access-qk6hh\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5\" (UID: \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" Nov 03 17:10:44.379743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.379640 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" Nov 03 17:10:44.964997 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:44.964969 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5"] Nov 03 17:10:45.220278 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.220182 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm"] Nov 03 17:10:45.243779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.243717 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" Nov 03 17:10:45.267783 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.267625 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm"] Nov 03 17:10:45.268035 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.268010 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"openshift-service-ca.crt\"" Nov 03 17:10:45.268246 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.268226 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"payload-processing-external-model-e2e--f8f5d575-dockercfg-zjb2n\"" Nov 03 17:10:45.268336 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.268318 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"kube-root-ca.crt\"" Nov 03 17:10:45.269073 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.269050 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"agc-cred-283408759ba7\"" Nov 03 17:10:45.269179 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.269125 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fdbf0c\"" Nov 03 17:10:45.269242 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.269190 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdbf0c\"/\"routing-overlay\"" Nov 03 17:10:45.430722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.423054 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7142049a-1792-4cd3-9528-297a5f90d234-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm\" (UID: \"7142049a-1792-4cd3-9528-297a5f90d234\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" Nov 03 17:10:45.430722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.423180 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn2rw\" (UniqueName: \"kubernetes.io/projected/7142049a-1792-4cd3-9528-297a5f90d234-kube-api-access-mn2rw\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm\" (UID: \"7142049a-1792-4cd3-9528-297a5f90d234\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" Nov 03 17:10:45.430722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.423303 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7142049a-1792-4cd3-9528-297a5f90d234-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm\" (UID: \"7142049a-1792-4cd3-9528-297a5f90d234\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" Nov 03 17:10:45.430722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.423361 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7142049a-1792-4cd3-9528-297a5f90d234-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm\" (UID: \"7142049a-1792-4cd3-9528-297a5f90d234\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" Nov 03 17:10:45.557725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.556878 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7142049a-1792-4cd3-9528-297a5f90d234-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm\" (UID: \"7142049a-1792-4cd3-9528-297a5f90d234\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" Nov 03 17:10:45.557725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.556994 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn2rw\" (UniqueName: \"kubernetes.io/projected/7142049a-1792-4cd3-9528-297a5f90d234-kube-api-access-mn2rw\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm\" (UID: \"7142049a-1792-4cd3-9528-297a5f90d234\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" Nov 03 17:10:45.557725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.557060 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7142049a-1792-4cd3-9528-297a5f90d234-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm\" (UID: \"7142049a-1792-4cd3-9528-297a5f90d234\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" Nov 03 17:10:45.557725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.557098 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7142049a-1792-4cd3-9528-297a5f90d234-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm\" (UID: \"7142049a-1792-4cd3-9528-297a5f90d234\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" Nov 03 17:10:45.565724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.558265 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7142049a-1792-4cd3-9528-297a5f90d234-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm\" (UID: \"7142049a-1792-4cd3-9528-297a5f90d234\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" Nov 03 17:10:45.565724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.559009 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7142049a-1792-4cd3-9528-297a5f90d234-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm\" (UID: \"7142049a-1792-4cd3-9528-297a5f90d234\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" Nov 03 17:10:45.626744 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.605618 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7142049a-1792-4cd3-9528-297a5f90d234-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm\" (UID: \"7142049a-1792-4cd3-9528-297a5f90d234\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" Nov 03 17:10:45.674924 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.671809 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn2rw\" (UniqueName: \"kubernetes.io/projected/7142049a-1792-4cd3-9528-297a5f90d234-kube-api-access-mn2rw\") pod \"payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm\" (UID: \"7142049a-1792-4cd3-9528-297a5f90d234\") " pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" Nov 03 17:10:45.720211 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.720170 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" event={"ID":"36f4f7e3-b29d-4ac3-aeff-03a5d19de191","Type":"ContainerStarted","Data":"8c380c4b26c82ebb740e1e472cdd50b8d4acc9d84803713410b8107079099b71"} Nov 03 17:10:45.720539 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.720495 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" event={"ID":"36f4f7e3-b29d-4ac3-aeff-03a5d19de191","Type":"ContainerStarted","Data":"1e7509c1d540836fcbecb01fb5a722c5248ba3abbae218cd758a0a56fc447a81"} Nov 03 17:10:45.720812 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.720781 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" Nov 03 17:10:45.806472 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.804233 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" podStartSLOduration=2.804209658 podStartE2EDuration="2.804209658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:45.788253956 +0000 UTC m=+3389.377253939" watchObservedRunningTime="2026-11-03 17:10:45.804209658 +0000 UTC m=+3389.393209628" Nov 03 17:10:45.878340 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:45.878245 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" Nov 03 17:10:46.318066 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:46.318040 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm"] Nov 03 17:10:46.321619 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:10:46.321590 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7142049a_1792_4cd3_9528_297a5f90d234.slice/crio-63b9d8bea81a196126f8a3934e14dcbc1ac084f8d73b25784604d577298c575f WatchSource:0}: Error finding container 63b9d8bea81a196126f8a3934e14dcbc1ac084f8d73b25784604d577298c575f: Status 404 returned error can't find the container with id 63b9d8bea81a196126f8a3934e14dcbc1ac084f8d73b25784604d577298c575f Nov 03 17:10:46.731958 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:46.731867 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" event={"ID":"7142049a-1792-4cd3-9528-297a5f90d234","Type":"ContainerStarted","Data":"6a94be7e7dcdce254f943730f3511cdbcdfe4fa29073fee3f3acae8595b51517"} Nov 03 17:10:46.732144 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:46.731964 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" event={"ID":"7142049a-1792-4cd3-9528-297a5f90d234","Type":"ContainerStarted","Data":"63b9d8bea81a196126f8a3934e14dcbc1ac084f8d73b25784604d577298c575f"} Nov 03 17:10:46.732265 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:46.732215 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" Nov 03 17:10:46.798752 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:46.793669 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" podStartSLOduration=1.793650406 podStartE2EDuration="1.793650406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:46.777290685 +0000 UTC m=+3390.366290677" watchObservedRunningTime="2026-11-03 17:10:46.793650406 +0000 UTC m=+3390.382650372" Nov 03 17:10:47.454111 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.451773 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx"] Nov 03 17:10:47.458245 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.458219 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" Nov 03 17:10:47.485727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.485669 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx"] Nov 03 17:10:47.492586 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.492556 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a0e27\"/\"payload-processing-external-model-plugins-e2e-extmod-a-3a0e27\"" Nov 03 17:10:47.492807 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.492615 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a0e27\"/\"openshift-service-ca.crt\"" Nov 03 17:10:47.492915 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.492675 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a0e27\"/\"kube-root-ca.crt\"" Nov 03 17:10:47.492915 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.492792 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3a0e27\"/\"routing-overlay\"" Nov 03 17:10:47.529587 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.529558 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a0e27\"/\"agc-cred-01e57ff28298\"" Nov 03 17:10:47.529937 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.529919 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3a0e27\"/\"payload-processing-external-model-e2e--8dd308cd-dockercfg-n5zj8\"" Nov 03 17:10:47.536435 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.536387 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55e20055-4a9c-4d8b-8d33-3d7583e3f469-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx\" (UID: \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" Nov 03 17:10:47.536597 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.536566 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55e20055-4a9c-4d8b-8d33-3d7583e3f469-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx\" (UID: \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" Nov 03 17:10:47.536648 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.536612 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hsww\" (UniqueName: \"kubernetes.io/projected/55e20055-4a9c-4d8b-8d33-3d7583e3f469-kube-api-access-5hsww\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx\" (UID: \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" Nov 03 17:10:47.536821 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.536800 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55e20055-4a9c-4d8b-8d33-3d7583e3f469-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx\" (UID: \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" Nov 03 17:10:47.649736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.638886 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55e20055-4a9c-4d8b-8d33-3d7583e3f469-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx\" (UID: \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" Nov 03 17:10:47.649736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.639141 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55e20055-4a9c-4d8b-8d33-3d7583e3f469-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx\" (UID: \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" Nov 03 17:10:47.649736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.639262 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55e20055-4a9c-4d8b-8d33-3d7583e3f469-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx\" (UID: \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" Nov 03 17:10:47.649736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.639301 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hsww\" (UniqueName: \"kubernetes.io/projected/55e20055-4a9c-4d8b-8d33-3d7583e3f469-kube-api-access-5hsww\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx\" (UID: \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" Nov 03 17:10:47.649736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.641051 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55e20055-4a9c-4d8b-8d33-3d7583e3f469-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx\" (UID: \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" Nov 03 17:10:47.658728 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.655323 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55e20055-4a9c-4d8b-8d33-3d7583e3f469-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx\" (UID: \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" Nov 03 17:10:47.677724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.674572 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55e20055-4a9c-4d8b-8d33-3d7583e3f469-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx\" (UID: \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" Nov 03 17:10:47.713768 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.712788 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hsww\" (UniqueName: \"kubernetes.io/projected/55e20055-4a9c-4d8b-8d33-3d7583e3f469-kube-api-access-5hsww\") pod \"payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx\" (UID: \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\") " pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" Nov 03 17:10:47.819977 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:47.819785 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" Nov 03 17:10:48.213058 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:10:48.213028 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e20055_4a9c_4d8b_8d33_3d7583e3f469.slice/crio-5022392eb13cbccd07140ec5a74740dd6928c3670f799497d005b07c7d5095e9 WatchSource:0}: Error finding container 5022392eb13cbccd07140ec5a74740dd6928c3670f799497d005b07c7d5095e9: Status 404 returned error can't find the container with id 5022392eb13cbccd07140ec5a74740dd6928c3670f799497d005b07c7d5095e9 Nov 03 17:10:48.213615 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.213523 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx"] Nov 03 17:10:48.598540 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.598506 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" Nov 03 17:10:48.611829 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.611794 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd"] Nov 03 17:10:48.618317 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.618291 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" Nov 03 17:10:48.634368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.634337 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-790111\"/\"payload-processing-external-model-plugins-e2e-extmod-b-790111\"" Nov 03 17:10:48.634529 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.634304 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-790111\"/\"openshift-service-ca.crt\"" Nov 03 17:10:48.634618 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.634561 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-790111\"/\"routing-overlay\"" Nov 03 17:10:48.634618 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.634562 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-790111\"/\"kube-root-ca.crt\"" Nov 03 17:10:48.634618 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.634604 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-790111\"/\"payload-processing-external-model-e2e--52ab0a1b-dockercfg-mbqtz\"" Nov 03 17:10:48.634775 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.634629 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-790111\"/\"agc-cred-363ec8af9c7a\"" Nov 03 17:10:48.636348 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.636329 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd"] Nov 03 17:10:48.637446 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.637418 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64232765-b0cc-498c-9041-048ed8285ad9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd\" (UID: \"64232765-b0cc-498c-9041-048ed8285ad9\") " pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" Nov 03 17:10:48.637674 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.637646 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64232765-b0cc-498c-9041-048ed8285ad9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd\" (UID: \"64232765-b0cc-498c-9041-048ed8285ad9\") " pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" Nov 03 17:10:48.637811 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.637714 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64232765-b0cc-498c-9041-048ed8285ad9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd\" (UID: \"64232765-b0cc-498c-9041-048ed8285ad9\") " pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" Nov 03 17:10:48.637867 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.637811 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxtrg\" (UniqueName: \"kubernetes.io/projected/64232765-b0cc-498c-9041-048ed8285ad9-kube-api-access-kxtrg\") pod \"payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd\" (UID: \"64232765-b0cc-498c-9041-048ed8285ad9\") " pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" Nov 03 17:10:48.739384 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.739342 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64232765-b0cc-498c-9041-048ed8285ad9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd\" (UID: \"64232765-b0cc-498c-9041-048ed8285ad9\") " pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" Nov 03 17:10:48.739551 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.739435 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64232765-b0cc-498c-9041-048ed8285ad9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd\" (UID: \"64232765-b0cc-498c-9041-048ed8285ad9\") " pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" Nov 03 17:10:48.739551 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.739476 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64232765-b0cc-498c-9041-048ed8285ad9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd\" (UID: \"64232765-b0cc-498c-9041-048ed8285ad9\") " pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" Nov 03 17:10:48.739551 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.739534 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxtrg\" (UniqueName: \"kubernetes.io/projected/64232765-b0cc-498c-9041-048ed8285ad9-kube-api-access-kxtrg\") pod \"payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd\" (UID: \"64232765-b0cc-498c-9041-048ed8285ad9\") " pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" Nov 03 17:10:48.740943 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.740914 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64232765-b0cc-498c-9041-048ed8285ad9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd\" (UID: \"64232765-b0cc-498c-9041-048ed8285ad9\") " pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" Nov 03 17:10:48.741452 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.741423 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64232765-b0cc-498c-9041-048ed8285ad9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd\" (UID: \"64232765-b0cc-498c-9041-048ed8285ad9\") " pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" Nov 03 17:10:48.743549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.743526 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64232765-b0cc-498c-9041-048ed8285ad9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd\" (UID: \"64232765-b0cc-498c-9041-048ed8285ad9\") " pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" Nov 03 17:10:48.764544 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.764514 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxtrg\" (UniqueName: \"kubernetes.io/projected/64232765-b0cc-498c-9041-048ed8285ad9-kube-api-access-kxtrg\") pod \"payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd\" (UID: \"64232765-b0cc-498c-9041-048ed8285ad9\") " pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" Nov 03 17:10:48.924935 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.924831 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" event={"ID":"55e20055-4a9c-4d8b-8d33-3d7583e3f469","Type":"ContainerStarted","Data":"c82788577110901a05e31c3d941cbffd5c77536bb5e996560707496fd1f51400"} Nov 03 17:10:48.925229 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.925189 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" event={"ID":"55e20055-4a9c-4d8b-8d33-3d7583e3f469","Type":"ContainerStarted","Data":"5022392eb13cbccd07140ec5a74740dd6928c3670f799497d005b07c7d5095e9"} Nov 03 17:10:48.926945 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.926924 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" Nov 03 17:10:48.940090 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.940058 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" Nov 03 17:10:49.004876 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:48.996256 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" podStartSLOduration=1.996234463 podStartE2EDuration="1.996234463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:48.972150776 +0000 UTC m=+3392.561150746" watchObservedRunningTime="2026-11-03 17:10:48.996234463 +0000 UTC m=+3392.585234433" Nov 03 17:10:49.354466 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:10:49.354439 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64232765_b0cc_498c_9041_048ed8285ad9.slice/crio-59b9f953720b3c4cc293f372ec85a8aa7c93e631900c33c3d01943ad7f5b5da4 WatchSource:0}: Error finding container 59b9f953720b3c4cc293f372ec85a8aa7c93e631900c33c3d01943ad7f5b5da4: Status 404 returned error can't find the container with id 59b9f953720b3c4cc293f372ec85a8aa7c93e631900c33c3d01943ad7f5b5da4 Nov 03 17:10:49.355097 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:49.355044 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd"] Nov 03 17:10:49.934404 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:49.934370 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" event={"ID":"64232765-b0cc-498c-9041-048ed8285ad9","Type":"ContainerStarted","Data":"8fcde897a2b8a0a65b2a9d6142c1def2a874eed399e9e2c761d47b775dad91b7"} Nov 03 17:10:49.934986 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:49.934426 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" event={"ID":"64232765-b0cc-498c-9041-048ed8285ad9","Type":"ContainerStarted","Data":"59b9f953720b3c4cc293f372ec85a8aa7c93e631900c33c3d01943ad7f5b5da4"} Nov 03 17:10:49.934986 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:49.934598 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" Nov 03 17:10:49.982099 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:49.982048 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" podStartSLOduration=1.982033708 podStartE2EDuration="1.982033708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:49.979914792 +0000 UTC m=+3393.568914762" watchObservedRunningTime="2026-11-03 17:10:49.982033708 +0000 UTC m=+3393.571033713" Nov 03 17:10:51.895270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:51.895228 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p"] Nov 03 17:10:51.901069 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:51.901042 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:51.930525 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:51.930491 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p"] Nov 03 17:10:51.948090 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:51.948060 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e0e63d-openshift-default-dockercfg-72z97\"" Nov 03 17:10:51.976328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:51.976295 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-podinfo\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:51.976526 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:51.976350 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpl4b\" (UniqueName: \"kubernetes.io/projected/54adffe4-51ee-41e0-8d1b-9d329330d512-kube-api-access-rpl4b\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:51.976526 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:51.976473 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-envoy\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:51.976526 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:51.976510 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-token\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:51.976784 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:51.976613 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-workload-certs\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:51.976784 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:51.976749 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54adffe4-51ee-41e0-8d1b-9d329330d512-istiod-ca-cert\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:51.976784 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:51.976781 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-credential-socket\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:51.976960 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:51.976898 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-data\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:51.976960 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:51.976944 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-workload-socket\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.078095 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.078059 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-envoy\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.078269 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.078114 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-token\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.078269 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.078171 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-workload-certs\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.078269 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.078246 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54adffe4-51ee-41e0-8d1b-9d329330d512-istiod-ca-cert\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.078269 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.078263 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-credential-socket\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.078500 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.078309 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-data\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.078500 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.078357 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-workload-socket\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.078500 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.078433 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-podinfo\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.078500 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.078477 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpl4b\" (UniqueName: \"kubernetes.io/projected/54adffe4-51ee-41e0-8d1b-9d329330d512-kube-api-access-rpl4b\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.078749 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.078671 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-credential-socket\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.078749 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.078734 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-workload-certs\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.079012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.078986 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-workload-socket\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.079117 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.078991 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-data\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.079203 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.079166 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54adffe4-51ee-41e0-8d1b-9d329330d512-istiod-ca-cert\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.080811 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.080789 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-envoy\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.081441 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.081415 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-podinfo\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.102788 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.102766 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-token\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.103049 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.103028 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpl4b\" (UniqueName: \"kubernetes.io/projected/54adffe4-51ee-41e0-8d1b-9d329330d512-kube-api-access-rpl4b\") pod \"e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.222374 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.222226 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:52.560501 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:10:52.560470 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54adffe4_51ee_41e0_8d1b_9d329330d512.slice/crio-9e14ca09b06db08c9797198e2ee98d790429050697d905ee52e1b584ed829bc4 WatchSource:0}: Error finding container 9e14ca09b06db08c9797198e2ee98d790429050697d905ee52e1b584ed829bc4: Status 404 returned error can't find the container with id 9e14ca09b06db08c9797198e2ee98d790429050697d905ee52e1b584ed829bc4 Nov 03 17:10:52.563804 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.563765 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:10:52.563910 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.563835 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:10:52.563910 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.563867 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:10:52.567990 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.567971 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p"] Nov 03 17:10:52.965133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.965035 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" event={"ID":"54adffe4-51ee-41e0-8d1b-9d329330d512","Type":"ContainerStarted","Data":"42c6ce51a1845fda5be1f27c45c86adea0805919d8da4a41fb9dc7a4d5598067"} Nov 03 17:10:52.965133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:52.965089 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" event={"ID":"54adffe4-51ee-41e0-8d1b-9d329330d512","Type":"ContainerStarted","Data":"9e14ca09b06db08c9797198e2ee98d790429050697d905ee52e1b584ed829bc4"} Nov 03 17:10:53.018354 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:53.018274 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" podStartSLOduration=2.018255674 podStartE2EDuration="2.018255674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:53.016300113 +0000 UTC m=+3396.605300081" watchObservedRunningTime="2026-11-03 17:10:53.018255674 +0000 UTC m=+3396.607255644" Nov 03 17:10:53.225995 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:53.225865 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:53.226559 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:53.226529 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Nov 03 17:10:53.226781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:53.226614 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" podUID="54adffe4-51ee-41e0-8d1b-9d329330d512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Nov 03 17:10:54.223249 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:54.223210 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Nov 03 17:10:54.223751 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:54.223272 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" podUID="54adffe4-51ee-41e0-8d1b-9d329330d512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Nov 03 17:10:55.226204 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:55.226163 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Nov 03 17:10:55.226637 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:55.226233 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" podUID="54adffe4-51ee-41e0-8d1b-9d329330d512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Nov 03 17:10:56.223303 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.223257 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Nov 03 17:10:56.223502 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.223324 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" podUID="54adffe4-51ee-41e0-8d1b-9d329330d512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Nov 03 17:10:56.390579 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.390540 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn"] Nov 03 17:10:56.395853 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.395831 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.430666 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.430631 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d6fcae-openshift-default-dockercfg-7qz7l\"" Nov 03 17:10:56.436276 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.436244 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn"] Nov 03 17:10:56.475447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.475348 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-istio-envoy\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.475447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.475410 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-istio-data\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.475684 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.475483 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea91c101-fb9f-4277-83d0-39229045f116-istio-token\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.475684 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.475640 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea91c101-fb9f-4277-83d0-39229045f116-istio-podinfo\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.475684 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.475680 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-workload-certs\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.475904 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.475867 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-workload-socket\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.475969 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.475919 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-credential-socket\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.476024 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.476001 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea91c101-fb9f-4277-83d0-39229045f116-istiod-ca-cert\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.476078 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.476048 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z7b8\" (UniqueName: \"kubernetes.io/projected/ea91c101-fb9f-4277-83d0-39229045f116-kube-api-access-4z7b8\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.577512 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.577478 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-workload-socket\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.577752 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.577553 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-credential-socket\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.577752 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.577625 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea91c101-fb9f-4277-83d0-39229045f116-istiod-ca-cert\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.577752 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.577690 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z7b8\" (UniqueName: \"kubernetes.io/projected/ea91c101-fb9f-4277-83d0-39229045f116-kube-api-access-4z7b8\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.577958 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.577850 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-istio-envoy\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.577958 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.577890 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-istio-data\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.577958 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.577925 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea91c101-fb9f-4277-83d0-39229045f116-istio-token\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.578115 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.577976 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea91c101-fb9f-4277-83d0-39229045f116-istio-podinfo\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.578115 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.577990 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-credential-socket\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.578115 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.578005 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-workload-socket\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.578115 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.578031 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-workload-certs\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.578280 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.578256 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-istio-data\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.578400 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.578378 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-workload-certs\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.578520 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.578499 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea91c101-fb9f-4277-83d0-39229045f116-istiod-ca-cert\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.580399 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.580370 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-istio-envoy\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.580512 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.580430 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea91c101-fb9f-4277-83d0-39229045f116-istio-podinfo\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.624877 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.624316 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea91c101-fb9f-4277-83d0-39229045f116-istio-token\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.630034 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.630001 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z7b8\" (UniqueName: \"kubernetes.io/projected/ea91c101-fb9f-4277-83d0-39229045f116-kube-api-access-4z7b8\") pod \"e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:56.736816 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.736730 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" Nov 03 17:10:56.739434 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:56.739412 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:57.125778 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:57.124469 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:10:57.125778 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:57.124603 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:10:57.125778 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:57.124670 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:10:57.128890 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:57.128851 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn"] Nov 03 17:10:57.228126 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:57.228092 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Nov 03 17:10:57.228331 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:57.228157 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" podUID="54adffe4-51ee-41e0-8d1b-9d329330d512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Nov 03 17:10:57.852915 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:57.852883 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" Nov 03 17:10:58.042266 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:58.042226 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" event={"ID":"ea91c101-fb9f-4277-83d0-39229045f116","Type":"ContainerStarted","Data":"48abbff8f9e2c04e08f62c52581d8bc0a50f458a753f1a9a2a063116b5e4e710"} Nov 03 17:10:58.042435 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:58.042295 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" event={"ID":"ea91c101-fb9f-4277-83d0-39229045f116","Type":"ContainerStarted","Data":"348d82c706e5ae246da8a5e0541e81502f385f8bb9bc60c6233111cac89355e6"} Nov 03 17:10:58.094050 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:58.093990 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" podStartSLOduration=2.093971995 podStartE2EDuration="2.093971995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:10:58.090595344 +0000 UTC m=+3401.679595313" watchObservedRunningTime="2026-11-03 17:10:58.093971995 +0000 UTC m=+3401.682971967" Nov 03 17:10:58.223286 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:58.223191 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Nov 03 17:10:58.223286 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:58.223259 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" podUID="54adffe4-51ee-41e0-8d1b-9d329330d512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Nov 03 17:10:58.739958 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:58.739868 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:10:58.742563 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:58.742534 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 17:10:58.742771 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:58.742589 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" podUID="ea91c101-fb9f-4277-83d0-39229045f116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 17:10:59.240469 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:59.240438 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:59.240974 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:59.240921 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:59.241683 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:59.241662 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:10:59.740013 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:59.739924 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 17:10:59.740013 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:10:59.739982 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" podUID="ea91c101-fb9f-4277-83d0-39229045f116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 17:11:00.150943 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:11:00.150904 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:11:00.740065 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:11:00.740031 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 17:11:00.740590 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:11:00.740092 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" podUID="ea91c101-fb9f-4277-83d0-39229045f116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 17:11:00.946829 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:11:00.946802 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" Nov 03 17:11:00.947022 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:11:00.946874 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" Nov 03 17:11:01.740514 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:11:01.740476 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 17:11:01.741056 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:11:01.740537 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" podUID="ea91c101-fb9f-4277-83d0-39229045f116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 17:11:02.747724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:11:02.747138 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 17:11:02.747724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:11:02.747216 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" podUID="ea91c101-fb9f-4277-83d0-39229045f116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 17:11:03.739869 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:11:03.739833 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body= Nov 03 17:11:03.740061 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:11:03.739900 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" podUID="ea91c101-fb9f-4277-83d0-39229045f116" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" Nov 03 17:11:04.745132 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:11:04.745100 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:11:04.745658 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:11:04.745638 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:11:04.746438 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:11:04.746419 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:11:30.131495 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:11:30.131459 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:11:36.733830 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:11:36.733798 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:12:00.178916 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:12:00.178825 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:12:17.611797 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:17.611750 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw"] Nov 03 17:12:22.612592 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:22.612551 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" podUID="32836a71-8b45-4d9c-be6a-995afe63f9df" containerName="payload-processing" containerID="cri-o://b6aa6880b8572ede6b388d660b3155ca3e7b6d722c563ee4fdd439f3be932c75" gracePeriod=25 Nov 03 17:12:22.910391 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:22.910365 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" Nov 03 17:12:22.932134 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:22.932103 4947 generic.go:356] "Generic (PLEG): container finished" podID="32836a71-8b45-4d9c-be6a-995afe63f9df" containerID="b6aa6880b8572ede6b388d660b3155ca3e7b6d722c563ee4fdd439f3be932c75" exitCode=0 Nov 03 17:12:22.932329 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:22.932150 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" Nov 03 17:12:22.932329 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:22.932181 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" event={"ID":"32836a71-8b45-4d9c-be6a-995afe63f9df","Type":"ContainerDied","Data":"b6aa6880b8572ede6b388d660b3155ca3e7b6d722c563ee4fdd439f3be932c75"} Nov 03 17:12:22.932329 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:22.932235 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw" event={"ID":"32836a71-8b45-4d9c-be6a-995afe63f9df","Type":"ContainerDied","Data":"0050cb4c7ad8dac602cb43fa23cb26b6392a5a318aa213f8afd9afb843d8f5ea"} Nov 03 17:12:22.932329 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:22.932257 4947 scope.go:122] "RemoveContainer" containerID="b6aa6880b8572ede6b388d660b3155ca3e7b6d722c563ee4fdd439f3be932c75" Nov 03 17:12:22.943204 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:22.943184 4947 scope.go:122] "RemoveContainer" containerID="b6aa6880b8572ede6b388d660b3155ca3e7b6d722c563ee4fdd439f3be932c75" Nov 03 17:12:22.943517 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:12:22.943487 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6aa6880b8572ede6b388d660b3155ca3e7b6d722c563ee4fdd439f3be932c75\": container with ID starting with b6aa6880b8572ede6b388d660b3155ca3e7b6d722c563ee4fdd439f3be932c75 not found: ID does not exist" containerID="b6aa6880b8572ede6b388d660b3155ca3e7b6d722c563ee4fdd439f3be932c75" Nov 03 17:12:22.943624 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:22.943522 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6aa6880b8572ede6b388d660b3155ca3e7b6d722c563ee4fdd439f3be932c75"} err="failed to get container status \"b6aa6880b8572ede6b388d660b3155ca3e7b6d722c563ee4fdd439f3be932c75\": rpc error: code = NotFound desc = could not find container \"b6aa6880b8572ede6b388d660b3155ca3e7b6d722c563ee4fdd439f3be932c75\": container with ID starting with b6aa6880b8572ede6b388d660b3155ca3e7b6d722c563ee4fdd439f3be932c75 not found: ID does not exist" Nov 03 17:12:23.027800 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:23.027762 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32836a71-8b45-4d9c-be6a-995afe63f9df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32836a71-8b45-4d9c-be6a-995afe63f9df-plugins-config-volume\") pod \"32836a71-8b45-4d9c-be6a-995afe63f9df\" (UID: \"32836a71-8b45-4d9c-be6a-995afe63f9df\") " Nov 03 17:12:23.028003 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:23.027953 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32836a71-8b45-4d9c-be6a-995afe63f9df-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/32836a71-8b45-4d9c-be6a-995afe63f9df-routing-overlay-volume\") pod \"32836a71-8b45-4d9c-be6a-995afe63f9df\" (UID: \"32836a71-8b45-4d9c-be6a-995afe63f9df\") " Nov 03 17:12:23.028082 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:23.028046 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32836a71-8b45-4d9c-be6a-995afe63f9df-provider-credentials\" (UniqueName: \"kubernetes.io/projected/32836a71-8b45-4d9c-be6a-995afe63f9df-provider-credentials\") pod \"32836a71-8b45-4d9c-be6a-995afe63f9df\" (UID: \"32836a71-8b45-4d9c-be6a-995afe63f9df\") " Nov 03 17:12:23.028143 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:23.028124 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32836a71-8b45-4d9c-be6a-995afe63f9df-kube-api-access-qgkgd\" (UniqueName: \"kubernetes.io/projected/32836a71-8b45-4d9c-be6a-995afe63f9df-kube-api-access-qgkgd\") pod \"32836a71-8b45-4d9c-be6a-995afe63f9df\" (UID: \"32836a71-8b45-4d9c-be6a-995afe63f9df\") " Nov 03 17:12:23.028301 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:23.028122 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32836a71-8b45-4d9c-be6a-995afe63f9df-plugins-config-volume" pod "32836a71-8b45-4d9c-be6a-995afe63f9df" (UID: "32836a71-8b45-4d9c-be6a-995afe63f9df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:12:23.028301 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:23.028295 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32836a71-8b45-4d9c-be6a-995afe63f9df-routing-overlay-volume" pod "32836a71-8b45-4d9c-be6a-995afe63f9df" (UID: "32836a71-8b45-4d9c-be6a-995afe63f9df"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:12:23.028845 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:23.028820 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/32836a71-8b45-4d9c-be6a-995afe63f9df-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:12:23.028951 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:23.028848 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32836a71-8b45-4d9c-be6a-995afe63f9df-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:12:23.030649 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:23.030628 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32836a71-8b45-4d9c-be6a-995afe63f9df-kube-api-access-qgkgd" pod "32836a71-8b45-4d9c-be6a-995afe63f9df" (UID: "32836a71-8b45-4d9c-be6a-995afe63f9df"). InnerVolumeSpecName "kube-api-access-qgkgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:12:23.030766 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:23.030657 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32836a71-8b45-4d9c-be6a-995afe63f9df-provider-credentials" pod "32836a71-8b45-4d9c-be6a-995afe63f9df" (UID: "32836a71-8b45-4d9c-be6a-995afe63f9df"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:12:23.131077 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:23.130991 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgkgd\" (UniqueName: \"kubernetes.io/projected/32836a71-8b45-4d9c-be6a-995afe63f9df-kube-api-access-qgkgd\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:12:23.131077 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:23.131024 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/32836a71-8b45-4d9c-be6a-995afe63f9df-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:12:23.285109 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:23.285067 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw"] Nov 03 17:12:23.296476 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:23.296431 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-85d692vgw"] Nov 03 17:12:24.568226 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:24.568185 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32836a71-8b45-4d9c-be6a-995afe63f9df" path="/var/lib/kubelet/pods/32836a71-8b45-4d9c-be6a-995afe63f9df/volumes" Nov 03 17:12:30.132077 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:12:30.132040 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:12:31.444360 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:31.444323 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:12:45.445870 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.445834 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl"] Nov 03 17:12:45.446915 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.446892 4947 cpu_manager.go:414] "containerMap: removing container" podUID="32836a71-8b45-4d9c-be6a-995afe63f9df" containerName="payload-processing" Nov 03 17:12:45.446915 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.446915 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32836a71-8b45-4d9c-be6a-995afe63f9df" containerName="payload-processing" Nov 03 17:12:45.447156 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.447141 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="32836a71-8b45-4d9c-be6a-995afe63f9df" containerName="payload-processing" Nov 03 17:12:45.452584 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.452561 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" Nov 03 17:12:45.468862 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.468839 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"agc-cred-464294770e39\"" Nov 03 17:12:45.474304 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.474281 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl"] Nov 03 17:12:45.482305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.482282 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"payload-processing-external-model-e2e--f0537159-dockercfg-vd526\"" Nov 03 17:12:45.482454 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.482282 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"openshift-service-ca.crt\"" Nov 03 17:12:45.482527 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.482295 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-da409b\"" Nov 03 17:12:45.482527 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.482317 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"routing-overlay\"" Nov 03 17:12:45.482527 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.482351 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-da409b\"/\"kube-root-ca.crt\"" Nov 03 17:12:45.506428 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.506391 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl\" (UID: \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" Nov 03 17:12:45.506597 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.506533 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl\" (UID: \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" Nov 03 17:12:45.506639 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.506620 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl\" (UID: \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" Nov 03 17:12:45.507027 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.507003 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wkkx\" (UniqueName: \"kubernetes.io/projected/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-kube-api-access-5wkkx\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl\" (UID: \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" Nov 03 17:12:45.609082 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.609042 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wkkx\" (UniqueName: \"kubernetes.io/projected/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-kube-api-access-5wkkx\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl\" (UID: \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" Nov 03 17:12:45.609384 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.609357 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl\" (UID: \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" Nov 03 17:12:45.609549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.609422 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl\" (UID: \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" Nov 03 17:12:45.609549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.609508 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl\" (UID: \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" Nov 03 17:12:45.612459 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.610080 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl\" (UID: \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" Nov 03 17:12:45.612459 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.611031 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl\" (UID: \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" Nov 03 17:12:45.612806 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.612787 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl\" (UID: \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" Nov 03 17:12:45.637261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.635397 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wkkx\" (UniqueName: \"kubernetes.io/projected/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-kube-api-access-5wkkx\") pod \"payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl\" (UID: \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\") " pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" Nov 03 17:12:45.771324 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:45.771005 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" Nov 03 17:12:46.185367 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:12:46.185337 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835b5ade_8a20_4c6c_ad78_2a72945bdcf2.slice/crio-0379c84930a622bf9f0c04495ae26d3e961dc565eaa14e65f34f5571eb81ceae WatchSource:0}: Error finding container 0379c84930a622bf9f0c04495ae26d3e961dc565eaa14e65f34f5571eb81ceae: Status 404 returned error can't find the container with id 0379c84930a622bf9f0c04495ae26d3e961dc565eaa14e65f34f5571eb81ceae Nov 03 17:12:46.186224 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:46.186200 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl"] Nov 03 17:12:46.361162 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:46.361127 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" event={"ID":"835b5ade-8a20-4c6c-ad78-2a72945bdcf2","Type":"ContainerStarted","Data":"b8893725bb07b5b717f4c0c5a33de29571af28394072d60df94d174fd02b63ad"} Nov 03 17:12:46.361350 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:46.361179 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" event={"ID":"835b5ade-8a20-4c6c-ad78-2a72945bdcf2","Type":"ContainerStarted","Data":"0379c84930a622bf9f0c04495ae26d3e961dc565eaa14e65f34f5571eb81ceae"} Nov 03 17:12:46.361350 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:46.361260 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" Nov 03 17:12:46.410803 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:46.410751 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" podStartSLOduration=1.410737082 podStartE2EDuration="1.410737082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:12:46.406773505 +0000 UTC m=+3509.995773473" watchObservedRunningTime="2026-11-03 17:12:46.410737082 +0000 UTC m=+3509.999737050" Nov 03 17:12:57.371282 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:12:57.371252 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" Nov 03 17:13:00.138176 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:13:00.138131 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:13:10.835824 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:10.835779 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl"] Nov 03 17:13:15.836719 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:15.836648 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" podUID="835b5ade-8a20-4c6c-ad78-2a72945bdcf2" containerName="payload-processing" containerID="cri-o://b8893725bb07b5b717f4c0c5a33de29571af28394072d60df94d174fd02b63ad" gracePeriod=25 Nov 03 17:13:16.143165 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.143135 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" Nov 03 17:13:16.255129 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.255096 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-routing-overlay-volume\") pod \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\" (UID: \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\") " Nov 03 17:13:16.255328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.255223 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-plugins-config-volume\") pod \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\" (UID: \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\") " Nov 03 17:13:16.255401 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.255386 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-kube-api-access-5wkkx\" (UniqueName: \"kubernetes.io/projected/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-kube-api-access-5wkkx\") pod \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\" (UID: \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\") " Nov 03 17:13:16.255479 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.255462 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-provider-credentials\") pod \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\" (UID: \"835b5ade-8a20-4c6c-ad78-2a72945bdcf2\") " Nov 03 17:13:16.255521 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.255489 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-routing-overlay-volume" pod "835b5ade-8a20-4c6c-ad78-2a72945bdcf2" (UID: "835b5ade-8a20-4c6c-ad78-2a72945bdcf2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:13:16.255613 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.255593 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-plugins-config-volume" pod "835b5ade-8a20-4c6c-ad78-2a72945bdcf2" (UID: "835b5ade-8a20-4c6c-ad78-2a72945bdcf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:13:16.256173 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.256153 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:13:16.256253 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.256174 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:13:16.257836 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.257804 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-provider-credentials" pod "835b5ade-8a20-4c6c-ad78-2a72945bdcf2" (UID: "835b5ade-8a20-4c6c-ad78-2a72945bdcf2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:13:16.258046 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.258027 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-kube-api-access-5wkkx" pod "835b5ade-8a20-4c6c-ad78-2a72945bdcf2" (UID: "835b5ade-8a20-4c6c-ad78-2a72945bdcf2"). InnerVolumeSpecName "kube-api-access-5wkkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:13:16.357685 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.357595 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wkkx\" (UniqueName: \"kubernetes.io/projected/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-kube-api-access-5wkkx\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:13:16.357685 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.357631 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/835b5ade-8a20-4c6c-ad78-2a72945bdcf2-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:13:16.632724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.632615 4947 generic.go:356] "Generic (PLEG): container finished" podID="835b5ade-8a20-4c6c-ad78-2a72945bdcf2" containerID="b8893725bb07b5b717f4c0c5a33de29571af28394072d60df94d174fd02b63ad" exitCode=0 Nov 03 17:13:16.632724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.632660 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" Nov 03 17:13:16.632951 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.632662 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" event={"ID":"835b5ade-8a20-4c6c-ad78-2a72945bdcf2","Type":"ContainerDied","Data":"b8893725bb07b5b717f4c0c5a33de29571af28394072d60df94d174fd02b63ad"} Nov 03 17:13:16.632951 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.632802 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl" event={"ID":"835b5ade-8a20-4c6c-ad78-2a72945bdcf2","Type":"ContainerDied","Data":"0379c84930a622bf9f0c04495ae26d3e961dc565eaa14e65f34f5571eb81ceae"} Nov 03 17:13:16.632951 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.632826 4947 scope.go:122] "RemoveContainer" containerID="b8893725bb07b5b717f4c0c5a33de29571af28394072d60df94d174fd02b63ad" Nov 03 17:13:16.643789 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.643762 4947 scope.go:122] "RemoveContainer" containerID="b8893725bb07b5b717f4c0c5a33de29571af28394072d60df94d174fd02b63ad" Nov 03 17:13:16.644126 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:13:16.644104 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8893725bb07b5b717f4c0c5a33de29571af28394072d60df94d174fd02b63ad\": container with ID starting with b8893725bb07b5b717f4c0c5a33de29571af28394072d60df94d174fd02b63ad not found: ID does not exist" containerID="b8893725bb07b5b717f4c0c5a33de29571af28394072d60df94d174fd02b63ad" Nov 03 17:13:16.644217 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.644134 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8893725bb07b5b717f4c0c5a33de29571af28394072d60df94d174fd02b63ad"} err="failed to get container status \"b8893725bb07b5b717f4c0c5a33de29571af28394072d60df94d174fd02b63ad\": rpc error: code = NotFound desc = could not find container \"b8893725bb07b5b717f4c0c5a33de29571af28394072d60df94d174fd02b63ad\": container with ID starting with b8893725bb07b5b717f4c0c5a33de29571af28394072d60df94d174fd02b63ad not found: ID does not exist" Nov 03 17:13:16.692828 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.692788 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl"] Nov 03 17:13:16.705403 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:16.705367 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-da409b/payload-processing-external-model-e2e-extmod-a-da409b-696br4lkl"] Nov 03 17:13:18.570949 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:18.570917 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="835b5ade-8a20-4c6c-ad78-2a72945bdcf2" path="/var/lib/kubelet/pods/835b5ade-8a20-4c6c-ad78-2a72945bdcf2/volumes" Nov 03 17:13:25.862348 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:25.862307 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:13:30.133359 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:13:30.133275 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:13:33.628513 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:33.628472 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d"] Nov 03 17:13:38.629747 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:38.629684 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" podUID="c6b80942-2b7a-4d66-a700-bbd1c642257e" containerName="payload-processing" containerID="cri-o://6dc38dea6b116263af1f54d5349a990f876154f950aa1ac56c7dc64b7f92e6b0" gracePeriod=25 Nov 03 17:13:38.872108 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:38.872069 4947 generic.go:356] "Generic (PLEG): container finished" podID="c6b80942-2b7a-4d66-a700-bbd1c642257e" containerID="6dc38dea6b116263af1f54d5349a990f876154f950aa1ac56c7dc64b7f92e6b0" exitCode=0 Nov 03 17:13:38.872306 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:38.872144 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" event={"ID":"c6b80942-2b7a-4d66-a700-bbd1c642257e","Type":"ContainerDied","Data":"6dc38dea6b116263af1f54d5349a990f876154f950aa1ac56c7dc64b7f92e6b0"} Nov 03 17:13:38.926679 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:38.926649 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" Nov 03 17:13:39.035625 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:39.035582 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6b80942-2b7a-4d66-a700-bbd1c642257e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6b80942-2b7a-4d66-a700-bbd1c642257e-routing-overlay-volume\") pod \"c6b80942-2b7a-4d66-a700-bbd1c642257e\" (UID: \"c6b80942-2b7a-4d66-a700-bbd1c642257e\") " Nov 03 17:13:39.035625 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:39.035632 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6b80942-2b7a-4d66-a700-bbd1c642257e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6b80942-2b7a-4d66-a700-bbd1c642257e-provider-credentials\") pod \"c6b80942-2b7a-4d66-a700-bbd1c642257e\" (UID: \"c6b80942-2b7a-4d66-a700-bbd1c642257e\") " Nov 03 17:13:39.035917 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:39.035710 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6b80942-2b7a-4d66-a700-bbd1c642257e-kube-api-access-97v9j\" (UniqueName: \"kubernetes.io/projected/c6b80942-2b7a-4d66-a700-bbd1c642257e-kube-api-access-97v9j\") pod \"c6b80942-2b7a-4d66-a700-bbd1c642257e\" (UID: \"c6b80942-2b7a-4d66-a700-bbd1c642257e\") " Nov 03 17:13:39.035917 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:39.035833 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6b80942-2b7a-4d66-a700-bbd1c642257e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b80942-2b7a-4d66-a700-bbd1c642257e-plugins-config-volume\") pod \"c6b80942-2b7a-4d66-a700-bbd1c642257e\" (UID: \"c6b80942-2b7a-4d66-a700-bbd1c642257e\") " Nov 03 17:13:39.036153 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:39.036122 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b80942-2b7a-4d66-a700-bbd1c642257e-routing-overlay-volume" pod "c6b80942-2b7a-4d66-a700-bbd1c642257e" (UID: "c6b80942-2b7a-4d66-a700-bbd1c642257e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:13:39.036302 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:39.036280 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b80942-2b7a-4d66-a700-bbd1c642257e-plugins-config-volume" pod "c6b80942-2b7a-4d66-a700-bbd1c642257e" (UID: "c6b80942-2b7a-4d66-a700-bbd1c642257e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:13:39.036598 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:39.036579 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c6b80942-2b7a-4d66-a700-bbd1c642257e-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:13:39.036671 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:39.036604 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6b80942-2b7a-4d66-a700-bbd1c642257e-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:13:39.038767 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:39.038736 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b80942-2b7a-4d66-a700-bbd1c642257e-provider-credentials" pod "c6b80942-2b7a-4d66-a700-bbd1c642257e" (UID: "c6b80942-2b7a-4d66-a700-bbd1c642257e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:13:39.038909 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:39.038780 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b80942-2b7a-4d66-a700-bbd1c642257e-kube-api-access-97v9j" pod "c6b80942-2b7a-4d66-a700-bbd1c642257e" (UID: "c6b80942-2b7a-4d66-a700-bbd1c642257e"). InnerVolumeSpecName "kube-api-access-97v9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:13:39.138691 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:39.138605 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c6b80942-2b7a-4d66-a700-bbd1c642257e-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:13:39.138691 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:39.138637 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97v9j\" (UniqueName: \"kubernetes.io/projected/c6b80942-2b7a-4d66-a700-bbd1c642257e-kube-api-access-97v9j\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:13:39.881383 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:39.881349 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" Nov 03 17:13:39.881383 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:39.881365 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d" event={"ID":"c6b80942-2b7a-4d66-a700-bbd1c642257e","Type":"ContainerDied","Data":"2aa6dd62c7bfe76f0c672682f1d66e663dbe3970897272ceffd08f8b2aaaf8a5"} Nov 03 17:13:39.881963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:39.881432 4947 scope.go:122] "RemoveContainer" containerID="6dc38dea6b116263af1f54d5349a990f876154f950aa1ac56c7dc64b7f92e6b0" Nov 03 17:13:39.939233 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:39.939197 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d"] Nov 03 17:13:39.952742 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:39.952689 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d6fcae/payload-processing-external-model-e2e-extmod-a-d6fcae-55c8sw79d"] Nov 03 17:13:40.569988 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:40.569951 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6b80942-2b7a-4d66-a700-bbd1c642257e" path="/var/lib/kubelet/pods/c6b80942-2b7a-4d66-a700-bbd1c642257e/volumes" Nov 03 17:13:50.606665 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.606630 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq"] Nov 03 17:13:50.609203 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.607451 4947 cpu_manager.go:414] "containerMap: removing container" podUID="c6b80942-2b7a-4d66-a700-bbd1c642257e" containerName="payload-processing" Nov 03 17:13:50.609203 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.607466 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b80942-2b7a-4d66-a700-bbd1c642257e" containerName="payload-processing" Nov 03 17:13:50.609203 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.607500 4947 cpu_manager.go:414] "containerMap: removing container" podUID="835b5ade-8a20-4c6c-ad78-2a72945bdcf2" containerName="payload-processing" Nov 03 17:13:50.609203 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.607507 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835b5ade-8a20-4c6c-ad78-2a72945bdcf2" containerName="payload-processing" Nov 03 17:13:50.609203 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.607620 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b80942-2b7a-4d66-a700-bbd1c642257e" containerName="payload-processing" Nov 03 17:13:50.609203 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.607636 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="835b5ade-8a20-4c6c-ad78-2a72945bdcf2" containerName="payload-processing" Nov 03 17:13:50.611754 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.611687 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" Nov 03 17:13:50.631962 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.631927 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq"] Nov 03 17:13:50.641062 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.641027 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 17:13:50.641062 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.641061 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 17:13:50.645011 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.644979 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 17:13:50.645011 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.644979 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f7d36ee398c5\"" Nov 03 17:13:50.645220 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.645020 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 17:13:50.646687 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.646661 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-p4x9n\"" Nov 03 17:13:50.701893 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.701853 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca9a5928-5d4c-4dd8-ba15-732ec581a967-plugins-config-volume\") pod \"payload-processing-external-model-7d6ff8c8df-6gstq\" (UID: \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\") " pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" Nov 03 17:13:50.702119 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.702103 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca9a5928-5d4c-4dd8-ba15-732ec581a967-routing-overlay-volume\") pod \"payload-processing-external-model-7d6ff8c8df-6gstq\" (UID: \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\") " pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" Nov 03 17:13:50.702314 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.702289 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca9a5928-5d4c-4dd8-ba15-732ec581a967-provider-credentials\") pod \"payload-processing-external-model-7d6ff8c8df-6gstq\" (UID: \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\") " pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" Nov 03 17:13:50.702401 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.702385 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjrfm\" (UniqueName: \"kubernetes.io/projected/ca9a5928-5d4c-4dd8-ba15-732ec581a967-kube-api-access-hjrfm\") pod \"payload-processing-external-model-7d6ff8c8df-6gstq\" (UID: \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\") " pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" Nov 03 17:13:50.823722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.823109 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca9a5928-5d4c-4dd8-ba15-732ec581a967-provider-credentials\") pod \"payload-processing-external-model-7d6ff8c8df-6gstq\" (UID: \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\") " pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" Nov 03 17:13:50.823722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.823223 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjrfm\" (UniqueName: \"kubernetes.io/projected/ca9a5928-5d4c-4dd8-ba15-732ec581a967-kube-api-access-hjrfm\") pod \"payload-processing-external-model-7d6ff8c8df-6gstq\" (UID: \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\") " pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" Nov 03 17:13:50.823722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.823256 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca9a5928-5d4c-4dd8-ba15-732ec581a967-plugins-config-volume\") pod \"payload-processing-external-model-7d6ff8c8df-6gstq\" (UID: \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\") " pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" Nov 03 17:13:50.823722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.823457 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca9a5928-5d4c-4dd8-ba15-732ec581a967-routing-overlay-volume\") pod \"payload-processing-external-model-7d6ff8c8df-6gstq\" (UID: \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\") " pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" Nov 03 17:13:50.836910 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.824257 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca9a5928-5d4c-4dd8-ba15-732ec581a967-routing-overlay-volume\") pod \"payload-processing-external-model-7d6ff8c8df-6gstq\" (UID: \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\") " pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" Nov 03 17:13:50.836910 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.824659 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca9a5928-5d4c-4dd8-ba15-732ec581a967-plugins-config-volume\") pod \"payload-processing-external-model-7d6ff8c8df-6gstq\" (UID: \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\") " pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" Nov 03 17:13:50.854064 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.853992 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca9a5928-5d4c-4dd8-ba15-732ec581a967-provider-credentials\") pod \"payload-processing-external-model-7d6ff8c8df-6gstq\" (UID: \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\") " pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" Nov 03 17:13:50.866752 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.866568 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjrfm\" (UniqueName: \"kubernetes.io/projected/ca9a5928-5d4c-4dd8-ba15-732ec581a967-kube-api-access-hjrfm\") pod \"payload-processing-external-model-7d6ff8c8df-6gstq\" (UID: \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\") " pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" Nov 03 17:13:50.927320 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:50.927013 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" Nov 03 17:13:51.339474 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:13:51.339439 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9a5928_5d4c_4dd8_ba15_732ec581a967.slice/crio-e2ef2b947d5a077f21f464a40021aba58aca59b8fe008dd8dec03afa913a4a7d WatchSource:0}: Error finding container e2ef2b947d5a077f21f464a40021aba58aca59b8fe008dd8dec03afa913a4a7d: Status 404 returned error can't find the container with id e2ef2b947d5a077f21f464a40021aba58aca59b8fe008dd8dec03afa913a4a7d Nov 03 17:13:51.339987 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:51.339938 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq"] Nov 03 17:13:52.091794 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:52.091755 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" event={"ID":"ca9a5928-5d4c-4dd8-ba15-732ec581a967","Type":"ContainerStarted","Data":"a17776b8cefe38c3afdcb3ba22fe05b3d32fd3cd9d5503b23a50700a4528cd2b"} Nov 03 17:13:52.092315 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:52.091822 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" event={"ID":"ca9a5928-5d4c-4dd8-ba15-732ec581a967","Type":"ContainerStarted","Data":"e2ef2b947d5a077f21f464a40021aba58aca59b8fe008dd8dec03afa913a4a7d"} Nov 03 17:13:52.092315 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:52.091880 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" Nov 03 17:13:52.139816 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:52.139754 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" podStartSLOduration=2.139736552 podStartE2EDuration="2.139736552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:13:52.136369995 +0000 UTC m=+3575.725369960" watchObservedRunningTime="2026-11-03 17:13:52.139736552 +0000 UTC m=+3575.728736520" Nov 03 17:13:56.406468 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:56.406433 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6"] Nov 03 17:13:56.411360 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:56.411336 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" Nov 03 17:13:56.428226 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:56.428201 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-1f83c0\"/\"agc-cred-1fa64834b355\"" Nov 03 17:13:56.448375 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:56.448336 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6"] Nov 03 17:13:56.575048 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:56.575005 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6njh\" (UniqueName: \"kubernetes.io/projected/f1ac846e-b262-45b8-a038-f6c40b04835e-kube-api-access-q6njh\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6\" (UID: \"f1ac846e-b262-45b8-a038-f6c40b04835e\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" Nov 03 17:13:56.575231 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:56.575110 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1ac846e-b262-45b8-a038-f6c40b04835e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6\" (UID: \"f1ac846e-b262-45b8-a038-f6c40b04835e\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" Nov 03 17:13:56.575231 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:56.575168 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1ac846e-b262-45b8-a038-f6c40b04835e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6\" (UID: \"f1ac846e-b262-45b8-a038-f6c40b04835e\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" Nov 03 17:13:56.575314 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:56.575295 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ac846e-b262-45b8-a038-f6c40b04835e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6\" (UID: \"f1ac846e-b262-45b8-a038-f6c40b04835e\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" Nov 03 17:13:56.676801 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:56.676674 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6njh\" (UniqueName: \"kubernetes.io/projected/f1ac846e-b262-45b8-a038-f6c40b04835e-kube-api-access-q6njh\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6\" (UID: \"f1ac846e-b262-45b8-a038-f6c40b04835e\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" Nov 03 17:13:56.676801 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:56.676802 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1ac846e-b262-45b8-a038-f6c40b04835e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6\" (UID: \"f1ac846e-b262-45b8-a038-f6c40b04835e\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" Nov 03 17:13:56.677078 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:56.676848 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1ac846e-b262-45b8-a038-f6c40b04835e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6\" (UID: \"f1ac846e-b262-45b8-a038-f6c40b04835e\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" Nov 03 17:13:56.677450 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:56.677422 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ac846e-b262-45b8-a038-f6c40b04835e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6\" (UID: \"f1ac846e-b262-45b8-a038-f6c40b04835e\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" Nov 03 17:13:56.677578 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:56.677503 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1ac846e-b262-45b8-a038-f6c40b04835e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6\" (UID: \"f1ac846e-b262-45b8-a038-f6c40b04835e\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" Nov 03 17:13:56.678101 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:56.678083 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ac846e-b262-45b8-a038-f6c40b04835e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6\" (UID: \"f1ac846e-b262-45b8-a038-f6c40b04835e\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" Nov 03 17:13:56.679759 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:56.679736 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1ac846e-b262-45b8-a038-f6c40b04835e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6\" (UID: \"f1ac846e-b262-45b8-a038-f6c40b04835e\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" Nov 03 17:13:56.709595 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:56.709549 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6njh\" (UniqueName: \"kubernetes.io/projected/f1ac846e-b262-45b8-a038-f6c40b04835e-kube-api-access-q6njh\") pod \"payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6\" (UID: \"f1ac846e-b262-45b8-a038-f6c40b04835e\") " pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" Nov 03 17:13:56.727615 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:56.727579 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" Nov 03 17:13:57.051658 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:13:57.051618 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ac846e_b262_45b8_a038_f6c40b04835e.slice/crio-c6acad687ed75a4da8e4cadddf9559ce6e3bb44b09cce2371f7efdb9916095b6 WatchSource:0}: Error finding container c6acad687ed75a4da8e4cadddf9559ce6e3bb44b09cce2371f7efdb9916095b6: Status 404 returned error can't find the container with id c6acad687ed75a4da8e4cadddf9559ce6e3bb44b09cce2371f7efdb9916095b6 Nov 03 17:13:57.052512 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:57.052488 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6"] Nov 03 17:13:57.187391 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:57.187356 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" event={"ID":"f1ac846e-b262-45b8-a038-f6c40b04835e","Type":"ContainerStarted","Data":"ff524d101015404ec969483daca24cbc3928f3c428becde4770df853f6f69029"} Nov 03 17:13:57.187593 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:57.187411 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" event={"ID":"f1ac846e-b262-45b8-a038-f6c40b04835e","Type":"ContainerStarted","Data":"c6acad687ed75a4da8e4cadddf9559ce6e3bb44b09cce2371f7efdb9916095b6"} Nov 03 17:13:57.187593 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:57.187575 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" Nov 03 17:13:57.248974 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:13:57.248920 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" podStartSLOduration=1.2489054880000001 podStartE2EDuration="1.248905488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:13:57.246561577 +0000 UTC m=+3580.835561544" watchObservedRunningTime="2026-11-03 17:13:57.248905488 +0000 UTC m=+3580.837905491" Nov 03 17:14:00.132661 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:14:00.132619 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:14:01.699628 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:01.699601 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6"] Nov 03 17:14:03.104045 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:03.104016 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" Nov 03 17:14:06.700828 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:06.700784 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" podUID="f1ac846e-b262-45b8-a038-f6c40b04835e" containerName="payload-processing" containerID="cri-o://ff524d101015404ec969483daca24cbc3928f3c428becde4770df853f6f69029" gracePeriod=25 Nov 03 17:14:07.039073 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.039045 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" Nov 03 17:14:07.177635 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.177597 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ac846e-b262-45b8-a038-f6c40b04835e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1ac846e-b262-45b8-a038-f6c40b04835e-provider-credentials\") pod \"f1ac846e-b262-45b8-a038-f6c40b04835e\" (UID: \"f1ac846e-b262-45b8-a038-f6c40b04835e\") " Nov 03 17:14:07.177882 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.177668 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ac846e-b262-45b8-a038-f6c40b04835e-kube-api-access-q6njh\" (UniqueName: \"kubernetes.io/projected/f1ac846e-b262-45b8-a038-f6c40b04835e-kube-api-access-q6njh\") pod \"f1ac846e-b262-45b8-a038-f6c40b04835e\" (UID: \"f1ac846e-b262-45b8-a038-f6c40b04835e\") " Nov 03 17:14:07.177882 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.177779 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ac846e-b262-45b8-a038-f6c40b04835e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1ac846e-b262-45b8-a038-f6c40b04835e-routing-overlay-volume\") pod \"f1ac846e-b262-45b8-a038-f6c40b04835e\" (UID: \"f1ac846e-b262-45b8-a038-f6c40b04835e\") " Nov 03 17:14:07.177882 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.177799 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ac846e-b262-45b8-a038-f6c40b04835e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ac846e-b262-45b8-a038-f6c40b04835e-plugins-config-volume\") pod \"f1ac846e-b262-45b8-a038-f6c40b04835e\" (UID: \"f1ac846e-b262-45b8-a038-f6c40b04835e\") " Nov 03 17:14:07.178313 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.178255 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ac846e-b262-45b8-a038-f6c40b04835e-plugins-config-volume" pod "f1ac846e-b262-45b8-a038-f6c40b04835e" (UID: "f1ac846e-b262-45b8-a038-f6c40b04835e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:07.178527 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.178443 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ac846e-b262-45b8-a038-f6c40b04835e-routing-overlay-volume" pod "f1ac846e-b262-45b8-a038-f6c40b04835e" (UID: "f1ac846e-b262-45b8-a038-f6c40b04835e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:07.178889 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.178870 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f1ac846e-b262-45b8-a038-f6c40b04835e-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:07.178980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.178893 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1ac846e-b262-45b8-a038-f6c40b04835e-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:07.180071 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.180045 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ac846e-b262-45b8-a038-f6c40b04835e-provider-credentials" pod "f1ac846e-b262-45b8-a038-f6c40b04835e" (UID: "f1ac846e-b262-45b8-a038-f6c40b04835e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:07.180192 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.180169 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ac846e-b262-45b8-a038-f6c40b04835e-kube-api-access-q6njh" pod "f1ac846e-b262-45b8-a038-f6c40b04835e" (UID: "f1ac846e-b262-45b8-a038-f6c40b04835e"). InnerVolumeSpecName "kube-api-access-q6njh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:07.281102 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.280998 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f1ac846e-b262-45b8-a038-f6c40b04835e-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:07.281102 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.281033 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6njh\" (UniqueName: \"kubernetes.io/projected/f1ac846e-b262-45b8-a038-f6c40b04835e-kube-api-access-q6njh\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:07.281459 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.281434 4947 generic.go:356] "Generic (PLEG): container finished" podID="f1ac846e-b262-45b8-a038-f6c40b04835e" containerID="ff524d101015404ec969483daca24cbc3928f3c428becde4770df853f6f69029" exitCode=0 Nov 03 17:14:07.281546 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.281481 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" event={"ID":"f1ac846e-b262-45b8-a038-f6c40b04835e","Type":"ContainerDied","Data":"ff524d101015404ec969483daca24cbc3928f3c428becde4770df853f6f69029"} Nov 03 17:14:07.281546 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.281500 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" Nov 03 17:14:07.281546 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.281523 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6" event={"ID":"f1ac846e-b262-45b8-a038-f6c40b04835e","Type":"ContainerDied","Data":"c6acad687ed75a4da8e4cadddf9559ce6e3bb44b09cce2371f7efdb9916095b6"} Nov 03 17:14:07.281546 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.281543 4947 scope.go:122] "RemoveContainer" containerID="ff524d101015404ec969483daca24cbc3928f3c428becde4770df853f6f69029" Nov 03 17:14:07.293404 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.293382 4947 scope.go:122] "RemoveContainer" containerID="ff524d101015404ec969483daca24cbc3928f3c428becde4770df853f6f69029" Nov 03 17:14:07.293752 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:14:07.293733 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff524d101015404ec969483daca24cbc3928f3c428becde4770df853f6f69029\": container with ID starting with ff524d101015404ec969483daca24cbc3928f3c428becde4770df853f6f69029 not found: ID does not exist" containerID="ff524d101015404ec969483daca24cbc3928f3c428becde4770df853f6f69029" Nov 03 17:14:07.293828 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.293771 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff524d101015404ec969483daca24cbc3928f3c428becde4770df853f6f69029"} err="failed to get container status \"ff524d101015404ec969483daca24cbc3928f3c428becde4770df853f6f69029\": rpc error: code = NotFound desc = could not find container \"ff524d101015404ec969483daca24cbc3928f3c428becde4770df853f6f69029\": container with ID starting with ff524d101015404ec969483daca24cbc3928f3c428becde4770df853f6f69029 not found: ID does not exist" Nov 03 17:14:07.341197 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.341166 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6"] Nov 03 17:14:07.345232 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:07.345204 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-f547llkt6"] Nov 03 17:14:08.568360 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:08.568326 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ac846e-b262-45b8-a038-f6c40b04835e" path="/var/lib/kubelet/pods/f1ac846e-b262-45b8-a038-f6c40b04835e/volumes" Nov 03 17:14:15.835853 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:15.835811 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb"] Nov 03 17:14:17.124036 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:17.124007 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:14:17.127491 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:17.127467 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:14:20.836495 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:20.836455 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" podUID="b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62" containerName="payload-processing" containerID="cri-o://4da11dd4771a590082cfec52fb377cf50de1099afca5bf1ca9acc30687df599b" gracePeriod=25 Nov 03 17:14:21.157372 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.157340 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" Nov 03 17:14:21.263190 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.263152 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-plugins-config-volume\") pod \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\" (UID: \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\") " Nov 03 17:14:21.263190 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.263193 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-routing-overlay-volume\") pod \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\" (UID: \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\") " Nov 03 17:14:21.263413 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.263286 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-kube-api-access-97zqn\" (UniqueName: \"kubernetes.io/projected/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-kube-api-access-97zqn\") pod \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\" (UID: \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\") " Nov 03 17:14:21.263413 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.263350 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-provider-credentials\") pod \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\" (UID: \"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62\") " Nov 03 17:14:21.263632 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.263606 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-routing-overlay-volume" pod "b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62" (UID: "b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:21.263683 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.263642 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-plugins-config-volume" pod "b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62" (UID: "b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:21.264056 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.264030 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:21.264196 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.264060 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:21.265913 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.265880 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-kube-api-access-97zqn" pod "b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62" (UID: "b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62"). InnerVolumeSpecName "kube-api-access-97zqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:21.266017 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.265965 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-provider-credentials" pod "b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62" (UID: "b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:21.365443 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.365407 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97zqn\" (UniqueName: \"kubernetes.io/projected/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-kube-api-access-97zqn\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:21.365443 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.365439 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:21.403516 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.403428 4947 generic.go:356] "Generic (PLEG): container finished" podID="b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62" containerID="4da11dd4771a590082cfec52fb377cf50de1099afca5bf1ca9acc30687df599b" exitCode=0 Nov 03 17:14:21.403516 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.403477 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" Nov 03 17:14:21.403798 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.403509 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" event={"ID":"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62","Type":"ContainerDied","Data":"4da11dd4771a590082cfec52fb377cf50de1099afca5bf1ca9acc30687df599b"} Nov 03 17:14:21.403798 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.403566 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb" event={"ID":"b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62","Type":"ContainerDied","Data":"e467c4962b50af9dddb1c176c66353091dacf5763260964b016acd452e34a6dc"} Nov 03 17:14:21.403798 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.403589 4947 scope.go:122] "RemoveContainer" containerID="4da11dd4771a590082cfec52fb377cf50de1099afca5bf1ca9acc30687df599b" Nov 03 17:14:21.415061 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.415038 4947 scope.go:122] "RemoveContainer" containerID="4da11dd4771a590082cfec52fb377cf50de1099afca5bf1ca9acc30687df599b" Nov 03 17:14:21.415342 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:14:21.415314 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da11dd4771a590082cfec52fb377cf50de1099afca5bf1ca9acc30687df599b\": container with ID starting with 4da11dd4771a590082cfec52fb377cf50de1099afca5bf1ca9acc30687df599b not found: ID does not exist" containerID="4da11dd4771a590082cfec52fb377cf50de1099afca5bf1ca9acc30687df599b" Nov 03 17:14:21.415392 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.415351 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da11dd4771a590082cfec52fb377cf50de1099afca5bf1ca9acc30687df599b"} err="failed to get container status \"4da11dd4771a590082cfec52fb377cf50de1099afca5bf1ca9acc30687df599b\": rpc error: code = NotFound desc = could not find container \"4da11dd4771a590082cfec52fb377cf50de1099afca5bf1ca9acc30687df599b\": container with ID starting with 4da11dd4771a590082cfec52fb377cf50de1099afca5bf1ca9acc30687df599b not found: ID does not exist" Nov 03 17:14:21.460931 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.460898 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb"] Nov 03 17:14:21.464930 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:21.464899 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-2492c4/payload-processing-external-model-e2e-extmod-a-2492c4-f4fdwhfwb"] Nov 03 17:14:22.403094 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:22.403065 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:14:22.584980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:22.584950 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62" path="/var/lib/kubelet/pods/b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62/volumes" Nov 03 17:14:24.756940 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:24.756900 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm"] Nov 03 17:14:26.151573 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:26.151533 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5"] Nov 03 17:14:26.785937 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:26.785890 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:26.810397 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:26.810356 4947 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": context deadline exceeded" Nov 03 17:14:26.811354 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:26.811322 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:26.889946 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:26.887951 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:26.919821 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:26.919766 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:26.921892 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:26.921855 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:26.969885 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:26.969832 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:26.971060 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:26.971006 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" podUID="2f4b9526-8df4-4d0f-a9bc-34824126f4ca" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:28.212499 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.212446 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-l2682 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:14:28.212985 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.212819 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" podUID="629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:14:28.494079 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.494047 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh"] Nov 03 17:14:28.494864 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.494848 4947 cpu_manager.go:414] "containerMap: removing container" podUID="f1ac846e-b262-45b8-a038-f6c40b04835e" containerName="payload-processing" Nov 03 17:14:28.494922 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.494866 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ac846e-b262-45b8-a038-f6c40b04835e" containerName="payload-processing" Nov 03 17:14:28.494922 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.494886 4947 cpu_manager.go:414] "containerMap: removing container" podUID="b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62" containerName="payload-processing" Nov 03 17:14:28.494922 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.494891 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62" containerName="payload-processing" Nov 03 17:14:28.495032 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.495023 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ac846e-b262-45b8-a038-f6c40b04835e" containerName="payload-processing" Nov 03 17:14:28.495093 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.495034 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e7fe0c-4df5-4f78-8dd1-9d6810d7be62" containerName="payload-processing" Nov 03 17:14:28.500332 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.500313 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" Nov 03 17:14:28.522589 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.522554 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh"] Nov 03 17:14:28.529605 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.529573 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"agc-cred-d25e830690f0\"" Nov 03 17:14:28.529797 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.529636 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"kube-root-ca.crt\"" Nov 03 17:14:28.530241 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.530220 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-6046c2\"" Nov 03 17:14:28.530297 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.530276 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"payload-processing-external-model-e2e--40ad01b7-dockercfg-5dkwn\"" Nov 03 17:14:28.532442 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.532422 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"openshift-service-ca.crt\"" Nov 03 17:14:28.532591 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.532458 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-6046c2\"/\"routing-overlay\"" Nov 03 17:14:28.588217 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.588182 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2da8a8c8-a680-45bb-967d-8b298f487c2b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh\" (UID: \"2da8a8c8-a680-45bb-967d-8b298f487c2b\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" Nov 03 17:14:28.588553 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.588524 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2da8a8c8-a680-45bb-967d-8b298f487c2b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh\" (UID: \"2da8a8c8-a680-45bb-967d-8b298f487c2b\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" Nov 03 17:14:28.588714 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.588647 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da8a8c8-a680-45bb-967d-8b298f487c2b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh\" (UID: \"2da8a8c8-a680-45bb-967d-8b298f487c2b\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" Nov 03 17:14:28.588868 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.588841 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbkn6\" (UniqueName: \"kubernetes.io/projected/2da8a8c8-a680-45bb-967d-8b298f487c2b-kube-api-access-xbkn6\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh\" (UID: \"2da8a8c8-a680-45bb-967d-8b298f487c2b\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" Nov 03 17:14:28.699745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.698026 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbkn6\" (UniqueName: \"kubernetes.io/projected/2da8a8c8-a680-45bb-967d-8b298f487c2b-kube-api-access-xbkn6\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh\" (UID: \"2da8a8c8-a680-45bb-967d-8b298f487c2b\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" Nov 03 17:14:28.699745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.698123 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2da8a8c8-a680-45bb-967d-8b298f487c2b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh\" (UID: \"2da8a8c8-a680-45bb-967d-8b298f487c2b\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" Nov 03 17:14:28.699745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.698368 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2da8a8c8-a680-45bb-967d-8b298f487c2b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh\" (UID: \"2da8a8c8-a680-45bb-967d-8b298f487c2b\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" Nov 03 17:14:28.699745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.698438 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da8a8c8-a680-45bb-967d-8b298f487c2b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh\" (UID: \"2da8a8c8-a680-45bb-967d-8b298f487c2b\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" Nov 03 17:14:28.724961 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.724682 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da8a8c8-a680-45bb-967d-8b298f487c2b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh\" (UID: \"2da8a8c8-a680-45bb-967d-8b298f487c2b\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" Nov 03 17:14:28.725921 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.725688 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2da8a8c8-a680-45bb-967d-8b298f487c2b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh\" (UID: \"2da8a8c8-a680-45bb-967d-8b298f487c2b\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" Nov 03 17:14:28.728511 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.728486 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2da8a8c8-a680-45bb-967d-8b298f487c2b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh\" (UID: \"2da8a8c8-a680-45bb-967d-8b298f487c2b\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" Nov 03 17:14:28.742556 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.742457 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbkn6\" (UniqueName: \"kubernetes.io/projected/2da8a8c8-a680-45bb-967d-8b298f487c2b-kube-api-access-xbkn6\") pod \"payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh\" (UID: \"2da8a8c8-a680-45bb-967d-8b298f487c2b\") " pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" Nov 03 17:14:28.817204 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:28.817112 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" Nov 03 17:14:29.218506 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:14:29.218466 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da8a8c8_a680_45bb_967d_8b298f487c2b.slice/crio-12912fc79e13d4147fdc8265681d8b9f7d419658f45fbfbb0c46dca2ce6c47e9 WatchSource:0}: Error finding container 12912fc79e13d4147fdc8265681d8b9f7d419658f45fbfbb0c46dca2ce6c47e9: Status 404 returned error can't find the container with id 12912fc79e13d4147fdc8265681d8b9f7d419658f45fbfbb0c46dca2ce6c47e9 Nov 03 17:14:29.224214 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:29.224183 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh"] Nov 03 17:14:29.757812 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:29.757771 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" podUID="7142049a-1792-4cd3-9528-297a5f90d234" containerName="payload-processing" containerID="cri-o://6a94be7e7dcdce254f943730f3511cdbcdfe4fa29073fee3f3acae8595b51517" gracePeriod=25 Nov 03 17:14:30.021970 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.018002 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" event={"ID":"2da8a8c8-a680-45bb-967d-8b298f487c2b","Type":"ContainerStarted","Data":"12186eb216b61143ba328a6134d0c70248a591e76f7307022c681aa94caf0b69"} Nov 03 17:14:30.021970 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.018067 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" event={"ID":"2da8a8c8-a680-45bb-967d-8b298f487c2b","Type":"ContainerStarted","Data":"12912fc79e13d4147fdc8265681d8b9f7d419658f45fbfbb0c46dca2ce6c47e9"} Nov 03 17:14:30.027155 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.027056 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" Nov 03 17:14:30.034810 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.034427 4947 generic.go:356] "Generic (PLEG): container finished" podID="7142049a-1792-4cd3-9528-297a5f90d234" containerID="6a94be7e7dcdce254f943730f3511cdbcdfe4fa29073fee3f3acae8595b51517" exitCode=0 Nov 03 17:14:30.034810 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.034506 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" event={"ID":"7142049a-1792-4cd3-9528-297a5f90d234","Type":"ContainerDied","Data":"6a94be7e7dcdce254f943730f3511cdbcdfe4fa29073fee3f3acae8595b51517"} Nov 03 17:14:30.092712 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.092630 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" podStartSLOduration=2.092615198 podStartE2EDuration="2.092615198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:14:30.09195501 +0000 UTC m=+3613.680954979" watchObservedRunningTime="2026-11-03 17:14:30.092615198 +0000 UTC m=+3613.681615165" Nov 03 17:14:30.157901 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:14:30.156478 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:14:30.342924 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.342896 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" Nov 03 17:14:30.417938 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.417901 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7142049a-1792-4cd3-9528-297a5f90d234-kube-api-access-mn2rw\" (UniqueName: \"kubernetes.io/projected/7142049a-1792-4cd3-9528-297a5f90d234-kube-api-access-mn2rw\") pod \"7142049a-1792-4cd3-9528-297a5f90d234\" (UID: \"7142049a-1792-4cd3-9528-297a5f90d234\") " Nov 03 17:14:30.418152 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.417985 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7142049a-1792-4cd3-9528-297a5f90d234-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7142049a-1792-4cd3-9528-297a5f90d234-routing-overlay-volume\") pod \"7142049a-1792-4cd3-9528-297a5f90d234\" (UID: \"7142049a-1792-4cd3-9528-297a5f90d234\") " Nov 03 17:14:30.418152 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.418126 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7142049a-1792-4cd3-9528-297a5f90d234-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7142049a-1792-4cd3-9528-297a5f90d234-provider-credentials\") pod \"7142049a-1792-4cd3-9528-297a5f90d234\" (UID: \"7142049a-1792-4cd3-9528-297a5f90d234\") " Nov 03 17:14:30.418399 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.418365 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7142049a-1792-4cd3-9528-297a5f90d234-routing-overlay-volume" pod "7142049a-1792-4cd3-9528-297a5f90d234" (UID: "7142049a-1792-4cd3-9528-297a5f90d234"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:30.418555 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.418422 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7142049a-1792-4cd3-9528-297a5f90d234-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7142049a-1792-4cd3-9528-297a5f90d234-plugins-config-volume\") pod \"7142049a-1792-4cd3-9528-297a5f90d234\" (UID: \"7142049a-1792-4cd3-9528-297a5f90d234\") " Nov 03 17:14:30.418940 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.418916 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7142049a-1792-4cd3-9528-297a5f90d234-plugins-config-volume" pod "7142049a-1792-4cd3-9528-297a5f90d234" (UID: "7142049a-1792-4cd3-9528-297a5f90d234"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:30.419405 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.419382 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7142049a-1792-4cd3-9528-297a5f90d234-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:30.419459 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.419418 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7142049a-1792-4cd3-9528-297a5f90d234-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:30.422331 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.422304 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7142049a-1792-4cd3-9528-297a5f90d234-provider-credentials" pod "7142049a-1792-4cd3-9528-297a5f90d234" (UID: "7142049a-1792-4cd3-9528-297a5f90d234"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:30.422856 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.422831 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7142049a-1792-4cd3-9528-297a5f90d234-kube-api-access-mn2rw" pod "7142049a-1792-4cd3-9528-297a5f90d234" (UID: "7142049a-1792-4cd3-9528-297a5f90d234"). InnerVolumeSpecName "kube-api-access-mn2rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:30.521096 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.521060 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn2rw\" (UniqueName: \"kubernetes.io/projected/7142049a-1792-4cd3-9528-297a5f90d234-kube-api-access-mn2rw\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:30.521096 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:30.521093 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7142049a-1792-4cd3-9528-297a5f90d234-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:31.044711 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.044669 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" Nov 03 17:14:31.044945 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.044717 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm" event={"ID":"7142049a-1792-4cd3-9528-297a5f90d234","Type":"ContainerDied","Data":"63b9d8bea81a196126f8a3934e14dcbc1ac084f8d73b25784604d577298c575f"} Nov 03 17:14:31.044945 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.044811 4947 scope.go:122] "RemoveContainer" containerID="6a94be7e7dcdce254f943730f3511cdbcdfe4fa29073fee3f3acae8595b51517" Nov 03 17:14:31.101996 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.101956 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm"] Nov 03 17:14:31.115732 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.115671 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdbf0c/payload-processing-external-model-e2e-extmod-b-fdbf0c-c56cbqgbm"] Nov 03 17:14:31.153153 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.153112 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" podUID="36f4f7e3-b29d-4ac3-aeff-03a5d19de191" containerName="payload-processing" containerID="cri-o://8c380c4b26c82ebb740e1e472cdd50b8d4acc9d84803713410b8107079099b71" gracePeriod=25 Nov 03 17:14:31.464776 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.464744 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" Nov 03 17:14:31.536846 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.536811 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-routing-overlay-volume\") pod \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\" (UID: \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\") " Nov 03 17:14:31.537036 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.536934 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-kube-api-access-qk6hh\" (UniqueName: \"kubernetes.io/projected/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-kube-api-access-qk6hh\") pod \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\" (UID: \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\") " Nov 03 17:14:31.537112 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.537050 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-plugins-config-volume\") pod \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\" (UID: \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\") " Nov 03 17:14:31.537174 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.537110 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-provider-credentials\" (UniqueName: \"kubernetes.io/projected/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-provider-credentials\") pod \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\" (UID: \"36f4f7e3-b29d-4ac3-aeff-03a5d19de191\") " Nov 03 17:14:31.537297 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.537253 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-routing-overlay-volume" pod "36f4f7e3-b29d-4ac3-aeff-03a5d19de191" (UID: "36f4f7e3-b29d-4ac3-aeff-03a5d19de191"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:31.537548 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.537520 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-plugins-config-volume" pod "36f4f7e3-b29d-4ac3-aeff-03a5d19de191" (UID: "36f4f7e3-b29d-4ac3-aeff-03a5d19de191"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:31.537898 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.537873 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:31.537976 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.537902 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:31.539474 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.539450 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-kube-api-access-qk6hh" pod "36f4f7e3-b29d-4ac3-aeff-03a5d19de191" (UID: "36f4f7e3-b29d-4ac3-aeff-03a5d19de191"). InnerVolumeSpecName "kube-api-access-qk6hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:31.539663 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.539645 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-provider-credentials" pod "36f4f7e3-b29d-4ac3-aeff-03a5d19de191" (UID: "36f4f7e3-b29d-4ac3-aeff-03a5d19de191"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:31.639591 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.639482 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:31.639591 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:31.639518 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk6hh\" (UniqueName: \"kubernetes.io/projected/36f4f7e3-b29d-4ac3-aeff-03a5d19de191-kube-api-access-qk6hh\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:32.054668 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:32.054634 4947 generic.go:356] "Generic (PLEG): container finished" podID="36f4f7e3-b29d-4ac3-aeff-03a5d19de191" containerID="8c380c4b26c82ebb740e1e472cdd50b8d4acc9d84803713410b8107079099b71" exitCode=0 Nov 03 17:14:32.054898 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:32.054687 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" Nov 03 17:14:32.054898 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:32.054744 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" event={"ID":"36f4f7e3-b29d-4ac3-aeff-03a5d19de191","Type":"ContainerDied","Data":"8c380c4b26c82ebb740e1e472cdd50b8d4acc9d84803713410b8107079099b71"} Nov 03 17:14:32.054898 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:32.054793 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5" event={"ID":"36f4f7e3-b29d-4ac3-aeff-03a5d19de191","Type":"ContainerDied","Data":"1e7509c1d540836fcbecb01fb5a722c5248ba3abbae218cd758a0a56fc447a81"} Nov 03 17:14:32.054898 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:32.054813 4947 scope.go:122] "RemoveContainer" containerID="8c380c4b26c82ebb740e1e472cdd50b8d4acc9d84803713410b8107079099b71" Nov 03 17:14:32.067637 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:32.067614 4947 scope.go:122] "RemoveContainer" containerID="8c380c4b26c82ebb740e1e472cdd50b8d4acc9d84803713410b8107079099b71" Nov 03 17:14:32.067991 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:14:32.067966 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c380c4b26c82ebb740e1e472cdd50b8d4acc9d84803713410b8107079099b71\": container with ID starting with 8c380c4b26c82ebb740e1e472cdd50b8d4acc9d84803713410b8107079099b71 not found: ID does not exist" containerID="8c380c4b26c82ebb740e1e472cdd50b8d4acc9d84803713410b8107079099b71" Nov 03 17:14:32.068057 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:32.068007 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c380c4b26c82ebb740e1e472cdd50b8d4acc9d84803713410b8107079099b71"} err="failed to get container status \"8c380c4b26c82ebb740e1e472cdd50b8d4acc9d84803713410b8107079099b71\": rpc error: code = NotFound desc = could not find container \"8c380c4b26c82ebb740e1e472cdd50b8d4acc9d84803713410b8107079099b71\": container with ID starting with 8c380c4b26c82ebb740e1e472cdd50b8d4acc9d84803713410b8107079099b71 not found: ID does not exist" Nov 03 17:14:32.113620 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:32.113582 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5"] Nov 03 17:14:32.129316 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:32.129276 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-1f83c0/payload-processing-external-model-e2e-extmod-a-1f83c0-595449zk5"] Nov 03 17:14:32.571407 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:32.571369 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f4f7e3-b29d-4ac3-aeff-03a5d19de191" path="/var/lib/kubelet/pods/36f4f7e3-b29d-4ac3-aeff-03a5d19de191/volumes" Nov 03 17:14:32.572158 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:32.572139 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7142049a-1792-4cd3-9528-297a5f90d234" path="/var/lib/kubelet/pods/7142049a-1792-4cd3-9528-297a5f90d234/volumes" Nov 03 17:14:33.915737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:33.915188 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77"] Nov 03 17:14:33.916342 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:33.916217 4947 cpu_manager.go:414] "containerMap: removing container" podUID="7142049a-1792-4cd3-9528-297a5f90d234" containerName="payload-processing" Nov 03 17:14:33.916342 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:33.916237 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7142049a-1792-4cd3-9528-297a5f90d234" containerName="payload-processing" Nov 03 17:14:33.916342 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:33.916257 4947 cpu_manager.go:414] "containerMap: removing container" podUID="36f4f7e3-b29d-4ac3-aeff-03a5d19de191" containerName="payload-processing" Nov 03 17:14:33.916342 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:33.916262 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f4f7e3-b29d-4ac3-aeff-03a5d19de191" containerName="payload-processing" Nov 03 17:14:33.916549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:33.916418 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="7142049a-1792-4cd3-9528-297a5f90d234" containerName="payload-processing" Nov 03 17:14:33.916549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:33.916430 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f4f7e3-b29d-4ac3-aeff-03a5d19de191" containerName="payload-processing" Nov 03 17:14:33.927747 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:33.924045 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:33.955728 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:33.955659 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77"] Nov 03 17:14:33.963204 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:33.960026 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-24cb6d-openshift-default-dockercfg-dz77t\"" Nov 03 17:14:34.007359 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.007320 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-istio-data\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.007588 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.007394 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a852b387-ca23-4e7a-b793-788feb393176-istio-token\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.007588 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.007553 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slsww\" (UniqueName: \"kubernetes.io/projected/a852b387-ca23-4e7a-b793-788feb393176-kube-api-access-slsww\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.007776 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.007600 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-istio-envoy\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.007776 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.007765 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-credential-socket\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.007883 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.007825 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-workload-certs\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.007883 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.007860 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-workload-socket\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.007979 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.007959 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a852b387-ca23-4e7a-b793-788feb393176-istio-podinfo\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.008035 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.008009 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a852b387-ca23-4e7a-b793-788feb393176-istiod-ca-cert\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.113971 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.113934 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-workload-certs\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.114923 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.114861 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-workload-socket\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.116688 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.116385 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a852b387-ca23-4e7a-b793-788feb393176-istio-podinfo\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.116688 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.116446 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a852b387-ca23-4e7a-b793-788feb393176-istiod-ca-cert\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.116688 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.116600 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-istio-data\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.116918 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.116738 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a852b387-ca23-4e7a-b793-788feb393176-istio-token\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.116918 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.116809 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slsww\" (UniqueName: \"kubernetes.io/projected/a852b387-ca23-4e7a-b793-788feb393176-kube-api-access-slsww\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.116918 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.116810 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-workload-socket\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.116918 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.116848 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-istio-envoy\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.116918 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.116902 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-credential-socket\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.118980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.117078 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-istio-data\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.118980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.117349 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-credential-socket\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.118980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.114583 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-workload-certs\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.118980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.118182 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a852b387-ca23-4e7a-b793-788feb393176-istiod-ca-cert\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.120470 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.120446 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a852b387-ca23-4e7a-b793-788feb393176-istio-podinfo\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.122370 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.122346 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-istio-envoy\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.156721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.149797 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a852b387-ca23-4e7a-b793-788feb393176-istio-token\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.179194 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.179105 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slsww\" (UniqueName: \"kubernetes.io/projected/a852b387-ca23-4e7a-b793-788feb393176-kube-api-access-slsww\") pod \"e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.252292 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.252216 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:34.583502 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:14:34.583455 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda852b387_ca23_4e7a_b793_788feb393176.slice/crio-902e7002ca49960b485b366c98ff554ac40e77a6b07505210b3a16b8b75a5474 WatchSource:0}: Error finding container 902e7002ca49960b485b366c98ff554ac40e77a6b07505210b3a16b8b75a5474: Status 404 returned error can't find the container with id 902e7002ca49960b485b366c98ff554ac40e77a6b07505210b3a16b8b75a5474 Nov 03 17:14:34.584601 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.584579 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77"] Nov 03 17:14:34.586363 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.586328 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:14:34.586448 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.586397 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:14:34.586448 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.586425 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:14:34.898854 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.898758 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh"] Nov 03 17:14:34.901569 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:34.901543 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" Nov 03 17:14:35.191280 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:35.191187 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" event={"ID":"a852b387-ca23-4e7a-b793-788feb393176","Type":"ContainerStarted","Data":"137cafd0e2519e9b29e6b9d0894445a8fd77e6b3aa47de07214a3dd16c740ae5"} Nov 03 17:14:35.191280 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:35.191243 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" event={"ID":"a852b387-ca23-4e7a-b793-788feb393176","Type":"ContainerStarted","Data":"902e7002ca49960b485b366c98ff554ac40e77a6b07505210b3a16b8b75a5474"} Nov 03 17:14:35.276001 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:35.275964 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:35.284618 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:35.279416 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" podStartSLOduration=2.279396131 podStartE2EDuration="2.279396131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:14:35.260174924 +0000 UTC m=+3618.849174892" watchObservedRunningTime="2026-11-03 17:14:35.279396131 +0000 UTC m=+3618.868396100" Nov 03 17:14:35.311725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:35.306733 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:36.202543 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:36.202444 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:36.204127 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:36.204095 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:14:39.616674 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:39.616638 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r"] Nov 03 17:14:39.621940 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:39.621911 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" Nov 03 17:14:39.638913 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:39.638884 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-24cb6d\"" Nov 03 17:14:39.655885 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:39.655846 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-24cb6d-dockercfg-sv2kp\"" Nov 03 17:14:39.660614 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:39.660571 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r"] Nov 03 17:14:39.765729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:39.753983 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqkkq\" (UniqueName: \"kubernetes.io/projected/458486ff-035b-44c6-853c-c6b071e5370f-kube-api-access-xqkkq\") pod \"payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r\" (UID: \"458486ff-035b-44c6-853c-c6b071e5370f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" Nov 03 17:14:39.765729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:39.754257 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/458486ff-035b-44c6-853c-c6b071e5370f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r\" (UID: \"458486ff-035b-44c6-853c-c6b071e5370f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" Nov 03 17:14:39.875779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:39.872190 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/458486ff-035b-44c6-853c-c6b071e5370f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r\" (UID: \"458486ff-035b-44c6-853c-c6b071e5370f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" Nov 03 17:14:39.875779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:39.872479 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqkkq\" (UniqueName: \"kubernetes.io/projected/458486ff-035b-44c6-853c-c6b071e5370f-kube-api-access-xqkkq\") pod \"payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r\" (UID: \"458486ff-035b-44c6-853c-c6b071e5370f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" Nov 03 17:14:39.920722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:39.910225 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/458486ff-035b-44c6-853c-c6b071e5370f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r\" (UID: \"458486ff-035b-44c6-853c-c6b071e5370f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" Nov 03 17:14:39.993504 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:39.946767 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" podUID="2da8a8c8-a680-45bb-967d-8b298f487c2b" containerName="payload-processing" containerID="cri-o://12186eb216b61143ba328a6134d0c70248a591e76f7307022c681aa94caf0b69" gracePeriod=25 Nov 03 17:14:40.004064 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.004037 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqkkq\" (UniqueName: \"kubernetes.io/projected/458486ff-035b-44c6-853c-c6b071e5370f-kube-api-access-xqkkq\") pod \"payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r\" (UID: \"458486ff-035b-44c6-853c-c6b071e5370f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" Nov 03 17:14:40.237908 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.237868 4947 generic.go:356] "Generic (PLEG): container finished" podID="2da8a8c8-a680-45bb-967d-8b298f487c2b" containerID="12186eb216b61143ba328a6134d0c70248a591e76f7307022c681aa94caf0b69" exitCode=0 Nov 03 17:14:40.238118 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.237904 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" event={"ID":"2da8a8c8-a680-45bb-967d-8b298f487c2b","Type":"ContainerDied","Data":"12186eb216b61143ba328a6134d0c70248a591e76f7307022c681aa94caf0b69"} Nov 03 17:14:40.239046 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.239026 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" Nov 03 17:14:40.335828 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.335798 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" Nov 03 17:14:40.436824 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.434360 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da8a8c8-a680-45bb-967d-8b298f487c2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da8a8c8-a680-45bb-967d-8b298f487c2b-plugins-config-volume\") pod \"2da8a8c8-a680-45bb-967d-8b298f487c2b\" (UID: \"2da8a8c8-a680-45bb-967d-8b298f487c2b\") " Nov 03 17:14:40.436824 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.434640 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2da8a8c8-a680-45bb-967d-8b298f487c2b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2da8a8c8-a680-45bb-967d-8b298f487c2b-routing-overlay-volume\") pod \"2da8a8c8-a680-45bb-967d-8b298f487c2b\" (UID: \"2da8a8c8-a680-45bb-967d-8b298f487c2b\") " Nov 03 17:14:40.436824 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.434728 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da8a8c8-a680-45bb-967d-8b298f487c2b-kube-api-access-xbkn6\" (UniqueName: \"kubernetes.io/projected/2da8a8c8-a680-45bb-967d-8b298f487c2b-kube-api-access-xbkn6\") pod \"2da8a8c8-a680-45bb-967d-8b298f487c2b\" (UID: \"2da8a8c8-a680-45bb-967d-8b298f487c2b\") " Nov 03 17:14:40.436824 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.434772 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2da8a8c8-a680-45bb-967d-8b298f487c2b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2da8a8c8-a680-45bb-967d-8b298f487c2b-provider-credentials\") pod \"2da8a8c8-a680-45bb-967d-8b298f487c2b\" (UID: \"2da8a8c8-a680-45bb-967d-8b298f487c2b\") " Nov 03 17:14:40.460748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.460576 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da8a8c8-a680-45bb-967d-8b298f487c2b-plugins-config-volume" pod "2da8a8c8-a680-45bb-967d-8b298f487c2b" (UID: "2da8a8c8-a680-45bb-967d-8b298f487c2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:40.461275 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.461242 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2da8a8c8-a680-45bb-967d-8b298f487c2b-routing-overlay-volume" pod "2da8a8c8-a680-45bb-967d-8b298f487c2b" (UID: "2da8a8c8-a680-45bb-967d-8b298f487c2b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:40.472507 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.472464 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da8a8c8-a680-45bb-967d-8b298f487c2b-kube-api-access-xbkn6" pod "2da8a8c8-a680-45bb-967d-8b298f487c2b" (UID: "2da8a8c8-a680-45bb-967d-8b298f487c2b"). InnerVolumeSpecName "kube-api-access-xbkn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:40.481073 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.481026 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2da8a8c8-a680-45bb-967d-8b298f487c2b-provider-credentials" pod "2da8a8c8-a680-45bb-967d-8b298f487c2b" (UID: "2da8a8c8-a680-45bb-967d-8b298f487c2b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:40.538249 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.538214 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbkn6\" (UniqueName: \"kubernetes.io/projected/2da8a8c8-a680-45bb-967d-8b298f487c2b-kube-api-access-xbkn6\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:40.538249 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.538246 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2da8a8c8-a680-45bb-967d-8b298f487c2b-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:40.538249 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.538256 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2da8a8c8-a680-45bb-967d-8b298f487c2b-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:40.538513 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.538264 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2da8a8c8-a680-45bb-967d-8b298f487c2b-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:40.571199 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.571179 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r"] Nov 03 17:14:40.824984 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.824950 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn"] Nov 03 17:14:40.826113 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.826086 4947 cpu_manager.go:414] "containerMap: removing container" podUID="2da8a8c8-a680-45bb-967d-8b298f487c2b" containerName="payload-processing" Nov 03 17:14:40.826113 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.826113 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2da8a8c8-a680-45bb-967d-8b298f487c2b" containerName="payload-processing" Nov 03 17:14:40.826329 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.826302 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="2da8a8c8-a680-45bb-967d-8b298f487c2b" containerName="payload-processing" Nov 03 17:14:40.830226 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.830206 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:40.869102 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.868878 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-44736d-openshift-default-dockercfg-f7xwg\"" Nov 03 17:14:40.874736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.874686 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn"] Nov 03 17:14:40.980722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.956420 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-workload-socket\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:40.981157 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.981126 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-podinfo\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:40.981306 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.981288 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-credential-socket\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:40.981431 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.981417 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istiod-ca-cert\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:40.981647 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.981632 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-workload-certs\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:40.981829 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.981814 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t47xx\" (UniqueName: \"kubernetes.io/projected/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-kube-api-access-t47xx\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:40.982046 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.982032 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-token\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.004723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:40.982156 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-data\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.005824 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.005790 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-envoy\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.110965 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.110866 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-workload-certs\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.110965 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.110951 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t47xx\" (UniqueName: \"kubernetes.io/projected/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-kube-api-access-t47xx\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.111231 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.111065 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-token\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.111231 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.111097 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-data\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.111231 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.111145 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-envoy\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.111401 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.111254 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-workload-socket\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.111401 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.111327 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-podinfo\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.111401 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.111360 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-credential-socket\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.111563 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.111401 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istiod-ca-cert\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.112347 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.112311 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istiod-ca-cert\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.112637 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.112615 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-workload-certs\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.114199 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.113378 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-data\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.119482 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.119455 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-workload-socket\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.168727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.119959 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-credential-socket\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.168727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.151362 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-podinfo\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.168727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.160157 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-token\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.170352 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.170308 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-envoy\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.178750 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.177625 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t47xx\" (UniqueName: \"kubernetes.io/projected/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-kube-api-access-t47xx\") pod \"e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.300724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.299679 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" event={"ID":"458486ff-035b-44c6-853c-c6b071e5370f","Type":"ContainerStarted","Data":"accac995b88726ac8f9715dfd000ebab0b3314a88bd2db9d39e9884288f78bfb"} Nov 03 17:14:41.300724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.299773 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" event={"ID":"458486ff-035b-44c6-853c-c6b071e5370f","Type":"ContainerStarted","Data":"223c1551c287249bac6463a152e85fed42bd2d681cac9c547c16144d6eddfaa0"} Nov 03 17:14:41.309194 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.301064 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" Nov 03 17:14:41.309194 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.303651 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" event={"ID":"2da8a8c8-a680-45bb-967d-8b298f487c2b","Type":"ContainerDied","Data":"12912fc79e13d4147fdc8265681d8b9f7d419658f45fbfbb0c46dca2ce6c47e9"} Nov 03 17:14:41.309194 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.303733 4947 scope.go:122] "RemoveContainer" containerID="12186eb216b61143ba328a6134d0c70248a591e76f7307022c681aa94caf0b69" Nov 03 17:14:41.309194 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.303921 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh" Nov 03 17:14:41.368156 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.368103 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" podStartSLOduration=2.368084735 podStartE2EDuration="2.368084735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:14:41.367977978 +0000 UTC m=+3624.956977947" watchObservedRunningTime="2026-11-03 17:14:41.368084735 +0000 UTC m=+3624.957084700" Nov 03 17:14:41.410804 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.410766 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh"] Nov 03 17:14:41.416356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.416321 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-6046c2/payload-processing-external-model-e2e-extmod-b-6046c2-6d54l8rhh"] Nov 03 17:14:41.450173 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.450137 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:41.834140 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:14:41.834104 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048bfe2e_c1d7_42c1_a75b_0bf2a25cb33d.slice/crio-6de2c686380ab14684ad7d06ea44f18b1d07235e4ada164c5dd34d5f4f7348fe WatchSource:0}: Error finding container 6de2c686380ab14684ad7d06ea44f18b1d07235e4ada164c5dd34d5f4f7348fe: Status 404 returned error can't find the container with id 6de2c686380ab14684ad7d06ea44f18b1d07235e4ada164c5dd34d5f4f7348fe Nov 03 17:14:41.835208 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.835182 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn"] Nov 03 17:14:41.837076 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.837035 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:14:41.837160 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.837116 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:14:41.837160 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:41.837148 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:14:42.314129 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:42.314092 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" event={"ID":"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d","Type":"ContainerStarted","Data":"fda76cd77932b7436280edfeff09a7edebabc200d13dfb1bbcc274088f8be176"} Nov 03 17:14:42.314318 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:42.314158 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" event={"ID":"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d","Type":"ContainerStarted","Data":"6de2c686380ab14684ad7d06ea44f18b1d07235e4ada164c5dd34d5f4f7348fe"} Nov 03 17:14:42.411858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:42.405669 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" podStartSLOduration=2.405631213 podStartE2EDuration="2.405631213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:14:42.38487279 +0000 UTC m=+3625.973872754" watchObservedRunningTime="2026-11-03 17:14:42.405631213 +0000 UTC m=+3625.994631182" Nov 03 17:14:42.451341 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:42.451305 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:42.453183 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:42.453155 4947 patch_prober.go:29] interesting pod/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 03 17:14:42.453360 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:42.453205 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" podUID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 03 17:14:42.591308 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:42.591229 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2da8a8c8-a680-45bb-967d-8b298f487c2b" path="/var/lib/kubelet/pods/2da8a8c8-a680-45bb-967d-8b298f487c2b/volumes" Nov 03 17:14:43.456397 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:43.456361 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:44.333315 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:44.333288 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:44.334236 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:44.334216 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:14:48.218277 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:48.218240 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq"] Nov 03 17:14:50.222222 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.222181 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55"] Nov 03 17:14:50.227503 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.227476 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.259233 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.259191 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fc40cc-openshift-default-dockercfg-hbqxj\"" Nov 03 17:14:50.276537 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.276500 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55"] Nov 03 17:14:50.370680 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.370645 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-workload-certs\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.370680 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.370693 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrdkl\" (UniqueName: \"kubernetes.io/projected/1508d21f-1858-4232-be82-bcda731862b9-kube-api-access-rrdkl\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.370966 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.370790 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-istio-envoy\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.370966 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.370931 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1508d21f-1858-4232-be82-bcda731862b9-istio-podinfo\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.371057 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.370992 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1508d21f-1858-4232-be82-bcda731862b9-istio-token\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.371109 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.371057 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-workload-socket\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.371146 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.371132 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1508d21f-1858-4232-be82-bcda731862b9-istiod-ca-cert\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.371188 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.371167 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-credential-socket\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.371285 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.371272 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-istio-data\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.482781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.477818 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-istio-data\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.509721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.495274 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-workload-certs\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.509721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.495555 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-istio-data\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.509721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.495582 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-workload-certs\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.509721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.495631 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrdkl\" (UniqueName: \"kubernetes.io/projected/1508d21f-1858-4232-be82-bcda731862b9-kube-api-access-rrdkl\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.509721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.495663 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-istio-envoy\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.509721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.495887 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1508d21f-1858-4232-be82-bcda731862b9-istio-podinfo\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.509721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.495992 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1508d21f-1858-4232-be82-bcda731862b9-istio-token\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.509721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.496074 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-workload-socket\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.509721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.496137 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1508d21f-1858-4232-be82-bcda731862b9-istiod-ca-cert\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.509721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.496180 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-credential-socket\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.509721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.496613 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-credential-socket\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.551733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.532069 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-workload-socket\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.551733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.532742 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1508d21f-1858-4232-be82-bcda731862b9-istiod-ca-cert\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.551733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.551666 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-istio-envoy\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.555718 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.552184 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1508d21f-1858-4232-be82-bcda731862b9-istio-podinfo\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.597727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.582415 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrdkl\" (UniqueName: \"kubernetes.io/projected/1508d21f-1858-4232-be82-bcda731862b9-kube-api-access-rrdkl\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.618752 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.618718 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1508d21f-1858-4232-be82-bcda731862b9-istio-token\") pod \"e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:50.845687 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:50.845651 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:51.174067 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:14:51.174033 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1508d21f_1858_4232_be82_bcda731862b9.slice/crio-cc13355cb6537493a68c5c76ae2349df40dc351856247edc2288c050568e7c6f WatchSource:0}: Error finding container cc13355cb6537493a68c5c76ae2349df40dc351856247edc2288c050568e7c6f: Status 404 returned error can't find the container with id cc13355cb6537493a68c5c76ae2349df40dc351856247edc2288c050568e7c6f Nov 03 17:14:51.175266 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:51.175242 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55"] Nov 03 17:14:51.177264 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:51.177227 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:14:51.177327 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:51.177301 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:14:51.177364 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:51.177331 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:14:51.395004 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:51.394969 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" event={"ID":"1508d21f-1858-4232-be82-bcda731862b9","Type":"ContainerStarted","Data":"03f58fdf817c9e0ca7390c901b4a4953c998c940c379b158ad2c2a1ea80e9245"} Nov 03 17:14:51.395511 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:51.395038 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" event={"ID":"1508d21f-1858-4232-be82-bcda731862b9","Type":"ContainerStarted","Data":"cc13355cb6537493a68c5c76ae2349df40dc351856247edc2288c050568e7c6f"} Nov 03 17:14:51.449379 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:51.449261 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" podStartSLOduration=1.449240884 podStartE2EDuration="1.449240884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:14:51.444725715 +0000 UTC m=+3635.033725683" watchObservedRunningTime="2026-11-03 17:14:51.449240884 +0000 UTC m=+3635.038240853" Nov 03 17:14:51.846124 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:51.846088 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:52.853464 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:52.853436 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:53.224786 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:53.220778 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" podUID="ca9a5928-5d4c-4dd8-ba15-732ec581a967" containerName="payload-processing" containerID="cri-o://a17776b8cefe38c3afdcb3ba22fe05b3d32fd3cd9d5503b23a50700a4528cd2b" gracePeriod=25 Nov 03 17:14:53.381721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:53.369261 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" Nov 03 17:14:53.479782 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:53.438023 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:53.479782 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:53.441572 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:14:53.726652 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:53.726622 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" Nov 03 17:14:53.802119 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:53.802086 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca9a5928-5d4c-4dd8-ba15-732ec581a967-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca9a5928-5d4c-4dd8-ba15-732ec581a967-routing-overlay-volume\") pod \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\" (UID: \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\") " Nov 03 17:14:53.802346 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:53.802177 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca9a5928-5d4c-4dd8-ba15-732ec581a967-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca9a5928-5d4c-4dd8-ba15-732ec581a967-provider-credentials\") pod \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\" (UID: \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\") " Nov 03 17:14:53.802346 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:53.802285 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca9a5928-5d4c-4dd8-ba15-732ec581a967-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca9a5928-5d4c-4dd8-ba15-732ec581a967-plugins-config-volume\") pod \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\" (UID: \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\") " Nov 03 17:14:53.802481 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:53.802354 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca9a5928-5d4c-4dd8-ba15-732ec581a967-kube-api-access-hjrfm\" (UniqueName: \"kubernetes.io/projected/ca9a5928-5d4c-4dd8-ba15-732ec581a967-kube-api-access-hjrfm\") pod \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\" (UID: \"ca9a5928-5d4c-4dd8-ba15-732ec581a967\") " Nov 03 17:14:53.802576 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:53.802539 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca9a5928-5d4c-4dd8-ba15-732ec581a967-routing-overlay-volume" pod "ca9a5928-5d4c-4dd8-ba15-732ec581a967" (UID: "ca9a5928-5d4c-4dd8-ba15-732ec581a967"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:53.802786 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:53.802765 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca9a5928-5d4c-4dd8-ba15-732ec581a967-plugins-config-volume" pod "ca9a5928-5d4c-4dd8-ba15-732ec581a967" (UID: "ca9a5928-5d4c-4dd8-ba15-732ec581a967"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:14:53.803062 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:53.803039 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca9a5928-5d4c-4dd8-ba15-732ec581a967-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:53.803149 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:53.803070 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca9a5928-5d4c-4dd8-ba15-732ec581a967-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:53.804722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:53.804686 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9a5928-5d4c-4dd8-ba15-732ec581a967-kube-api-access-hjrfm" pod "ca9a5928-5d4c-4dd8-ba15-732ec581a967" (UID: "ca9a5928-5d4c-4dd8-ba15-732ec581a967"). InnerVolumeSpecName "kube-api-access-hjrfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:53.804970 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:53.804954 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9a5928-5d4c-4dd8-ba15-732ec581a967-provider-credentials" pod "ca9a5928-5d4c-4dd8-ba15-732ec581a967" (UID: "ca9a5928-5d4c-4dd8-ba15-732ec581a967"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:14:53.925402 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:53.925361 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjrfm\" (UniqueName: \"kubernetes.io/projected/ca9a5928-5d4c-4dd8-ba15-732ec581a967-kube-api-access-hjrfm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:53.925916 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:53.925401 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca9a5928-5d4c-4dd8-ba15-732ec581a967-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:14:54.446000 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:54.445962 4947 generic.go:356] "Generic (PLEG): container finished" podID="ca9a5928-5d4c-4dd8-ba15-732ec581a967" containerID="a17776b8cefe38c3afdcb3ba22fe05b3d32fd3cd9d5503b23a50700a4528cd2b" exitCode=0 Nov 03 17:14:54.446175 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:54.446000 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" event={"ID":"ca9a5928-5d4c-4dd8-ba15-732ec581a967","Type":"ContainerDied","Data":"a17776b8cefe38c3afdcb3ba22fe05b3d32fd3cd9d5503b23a50700a4528cd2b"} Nov 03 17:14:54.446175 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:54.446022 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" Nov 03 17:14:54.446175 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:54.446059 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq" event={"ID":"ca9a5928-5d4c-4dd8-ba15-732ec581a967","Type":"ContainerDied","Data":"e2ef2b947d5a077f21f464a40021aba58aca59b8fe008dd8dec03afa913a4a7d"} Nov 03 17:14:54.446175 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:54.446082 4947 scope.go:122] "RemoveContainer" containerID="a17776b8cefe38c3afdcb3ba22fe05b3d32fd3cd9d5503b23a50700a4528cd2b" Nov 03 17:14:54.458169 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:54.458146 4947 scope.go:122] "RemoveContainer" containerID="a17776b8cefe38c3afdcb3ba22fe05b3d32fd3cd9d5503b23a50700a4528cd2b" Nov 03 17:14:54.458459 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:14:54.458435 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17776b8cefe38c3afdcb3ba22fe05b3d32fd3cd9d5503b23a50700a4528cd2b\": container with ID starting with a17776b8cefe38c3afdcb3ba22fe05b3d32fd3cd9d5503b23a50700a4528cd2b not found: ID does not exist" containerID="a17776b8cefe38c3afdcb3ba22fe05b3d32fd3cd9d5503b23a50700a4528cd2b" Nov 03 17:14:54.458541 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:54.458465 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17776b8cefe38c3afdcb3ba22fe05b3d32fd3cd9d5503b23a50700a4528cd2b"} err="failed to get container status \"a17776b8cefe38c3afdcb3ba22fe05b3d32fd3cd9d5503b23a50700a4528cd2b\": rpc error: code = NotFound desc = could not find container \"a17776b8cefe38c3afdcb3ba22fe05b3d32fd3cd9d5503b23a50700a4528cd2b\": container with ID starting with a17776b8cefe38c3afdcb3ba22fe05b3d32fd3cd9d5503b23a50700a4528cd2b not found: ID does not exist" Nov 03 17:14:54.508585 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:54.508547 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq"] Nov 03 17:14:54.512723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:54.512679 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7d6ff8c8df-6gstq"] Nov 03 17:14:54.567616 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:54.567581 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca9a5928-5d4c-4dd8-ba15-732ec581a967" path="/var/lib/kubelet/pods/ca9a5928-5d4c-4dd8-ba15-732ec581a967/volumes" Nov 03 17:14:55.009951 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:55.009916 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx"] Nov 03 17:14:56.330159 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:56.330129 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd"] Nov 03 17:14:58.112108 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.112007 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv"] Nov 03 17:14:58.128972 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.128937 4947 cpu_manager.go:414] "containerMap: removing container" podUID="ca9a5928-5d4c-4dd8-ba15-732ec581a967" containerName="payload-processing" Nov 03 17:14:58.151507 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.151477 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca9a5928-5d4c-4dd8-ba15-732ec581a967" containerName="payload-processing" Nov 03 17:14:58.152116 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.152095 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca9a5928-5d4c-4dd8-ba15-732ec581a967" containerName="payload-processing" Nov 03 17:14:58.425013 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.423600 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" Nov 03 17:14:58.432087 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.427748 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6"] Nov 03 17:14:58.453254 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.453222 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-fc40cc\"" Nov 03 17:14:58.472827 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.472769 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" Nov 03 17:14:58.493393 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.491119 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-fc40cc-dockercfg-r4s2b\"" Nov 03 17:14:58.495117 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.495073 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv"] Nov 03 17:14:58.505230 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.502708 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6"] Nov 03 17:14:58.522815 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.522781 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47ms\" (UniqueName: \"kubernetes.io/projected/83746e7d-ec09-4142-b46b-38fa1471cc4a-kube-api-access-x47ms\") pod \"payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv\" (UID: \"83746e7d-ec09-4142-b46b-38fa1471cc4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" Nov 03 17:14:58.522951 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.522861 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c3d1351-007d-4fe5-be27-d868a8655c32-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6\" (UID: \"8c3d1351-007d-4fe5-be27-d868a8655c32\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" Nov 03 17:14:58.522951 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.522915 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nkdc\" (UniqueName: \"kubernetes.io/projected/8c3d1351-007d-4fe5-be27-d868a8655c32-kube-api-access-5nkdc\") pod \"payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6\" (UID: \"8c3d1351-007d-4fe5-be27-d868a8655c32\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" Nov 03 17:14:58.523044 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.522960 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83746e7d-ec09-4142-b46b-38fa1471cc4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv\" (UID: \"83746e7d-ec09-4142-b46b-38fa1471cc4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" Nov 03 17:14:58.624977 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.624934 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nkdc\" (UniqueName: \"kubernetes.io/projected/8c3d1351-007d-4fe5-be27-d868a8655c32-kube-api-access-5nkdc\") pod \"payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6\" (UID: \"8c3d1351-007d-4fe5-be27-d868a8655c32\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" Nov 03 17:14:58.625157 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.625000 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83746e7d-ec09-4142-b46b-38fa1471cc4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv\" (UID: \"83746e7d-ec09-4142-b46b-38fa1471cc4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" Nov 03 17:14:58.625293 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.625281 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47ms\" (UniqueName: \"kubernetes.io/projected/83746e7d-ec09-4142-b46b-38fa1471cc4a-kube-api-access-x47ms\") pod \"payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv\" (UID: \"83746e7d-ec09-4142-b46b-38fa1471cc4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" Nov 03 17:14:58.625344 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.625330 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c3d1351-007d-4fe5-be27-d868a8655c32-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6\" (UID: \"8c3d1351-007d-4fe5-be27-d868a8655c32\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" Nov 03 17:14:58.625899 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.625877 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83746e7d-ec09-4142-b46b-38fa1471cc4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv\" (UID: \"83746e7d-ec09-4142-b46b-38fa1471cc4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" Nov 03 17:14:58.626046 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.626029 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c3d1351-007d-4fe5-be27-d868a8655c32-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6\" (UID: \"8c3d1351-007d-4fe5-be27-d868a8655c32\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" Nov 03 17:14:58.650012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.649977 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nkdc\" (UniqueName: \"kubernetes.io/projected/8c3d1351-007d-4fe5-be27-d868a8655c32-kube-api-access-5nkdc\") pod \"payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6\" (UID: \"8c3d1351-007d-4fe5-be27-d868a8655c32\") " pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" Nov 03 17:14:58.650012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.649993 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47ms\" (UniqueName: \"kubernetes.io/projected/83746e7d-ec09-4142-b46b-38fa1471cc4a-kube-api-access-x47ms\") pod \"payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv\" (UID: \"83746e7d-ec09-4142-b46b-38fa1471cc4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" Nov 03 17:14:58.807986 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.807957 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" Nov 03 17:14:58.825558 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:58.825417 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" Nov 03 17:14:59.122417 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:14:59.122386 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83746e7d_ec09_4142_b46b_38fa1471cc4a.slice/crio-2557f44df31e7220457498fc55fcfdac0923b21e0bfdf1f5daf2ab0979d2c82c WatchSource:0}: Error finding container 2557f44df31e7220457498fc55fcfdac0923b21e0bfdf1f5daf2ab0979d2c82c: Status 404 returned error can't find the container with id 2557f44df31e7220457498fc55fcfdac0923b21e0bfdf1f5daf2ab0979d2c82c Nov 03 17:14:59.125839 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:59.125814 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv"] Nov 03 17:14:59.140629 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:14:59.140599 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3d1351_007d_4fe5_be27_d868a8655c32.slice/crio-4e399d74862b94d36240238be54b0f366c4210435b2afce14b16ddcdcdd2b1d5 WatchSource:0}: Error finding container 4e399d74862b94d36240238be54b0f366c4210435b2afce14b16ddcdcdd2b1d5: Status 404 returned error can't find the container with id 4e399d74862b94d36240238be54b0f366c4210435b2afce14b16ddcdcdd2b1d5 Nov 03 17:14:59.141132 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:59.141109 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6"] Nov 03 17:14:59.531068 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:59.531026 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" event={"ID":"83746e7d-ec09-4142-b46b-38fa1471cc4a","Type":"ContainerStarted","Data":"d43a8352373cafbdc8f158fa1686efd34aad670cf38d224b6b89959f59c91c19"} Nov 03 17:14:59.531303 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:59.531094 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" event={"ID":"83746e7d-ec09-4142-b46b-38fa1471cc4a","Type":"ContainerStarted","Data":"2557f44df31e7220457498fc55fcfdac0923b21e0bfdf1f5daf2ab0979d2c82c"} Nov 03 17:14:59.531303 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:59.531127 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" Nov 03 17:14:59.532838 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:59.532809 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" event={"ID":"8c3d1351-007d-4fe5-be27-d868a8655c32","Type":"ContainerStarted","Data":"3e8600e18ede2dff2a5d040b8efb4857acdced260c0001f074d83bf94dfbda59"} Nov 03 17:14:59.532966 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:59.532854 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" event={"ID":"8c3d1351-007d-4fe5-be27-d868a8655c32","Type":"ContainerStarted","Data":"4e399d74862b94d36240238be54b0f366c4210435b2afce14b16ddcdcdd2b1d5"} Nov 03 17:14:59.532966 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:59.532882 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" Nov 03 17:14:59.577998 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:59.577931 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" podStartSLOduration=1.5779162759999998 podStartE2EDuration="1.577916276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:14:59.575007611 +0000 UTC m=+3643.164007580" watchObservedRunningTime="2026-11-03 17:14:59.577916276 +0000 UTC m=+3643.166916248" Nov 03 17:14:59.619144 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:14:59.619089 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" podStartSLOduration=1.619069368 podStartE2EDuration="1.619069368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:14:59.618366395 +0000 UTC m=+3643.207366362" watchObservedRunningTime="2026-11-03 17:14:59.619069368 +0000 UTC m=+3643.208069337" Nov 03 17:15:00.010572 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.010530 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" podUID="55e20055-4a9c-4d8b-8d33-3d7583e3f469" containerName="payload-processing" containerID="cri-o://c82788577110901a05e31c3d941cbffd5c77536bb5e996560707496fd1f51400" gracePeriod=25 Nov 03 17:15:00.134219 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:00.134177 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:15:00.311662 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.311614 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b"] Nov 03 17:15:00.318127 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.318094 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" Nov 03 17:15:00.319430 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.319396 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk"] Nov 03 17:15:00.325074 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.325050 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" Nov 03 17:15:00.332688 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.332657 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" Nov 03 17:15:00.345768 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.343619 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b"] Nov 03 17:15:00.349819 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.349789 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk"] Nov 03 17:15:00.353902 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.353869 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:15:00.355173 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.355148 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr"] Nov 03 17:15:00.356587 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.356562 4947 cpu_manager.go:414] "containerMap: removing container" podUID="55e20055-4a9c-4d8b-8d33-3d7583e3f469" containerName="payload-processing" Nov 03 17:15:00.356587 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.356586 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e20055-4a9c-4d8b-8d33-3d7583e3f469" containerName="payload-processing" Nov 03 17:15:00.356922 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.356903 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e20055-4a9c-4d8b-8d33-3d7583e3f469" containerName="payload-processing" Nov 03 17:15:00.362486 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.362465 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl"] Nov 03 17:15:00.362633 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.362596 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" Nov 03 17:15:00.375929 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.372264 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9"] Nov 03 17:15:00.375929 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.373007 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" Nov 03 17:15:00.385945 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.385906 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr"] Nov 03 17:15:00.386100 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.386024 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" Nov 03 17:15:00.392188 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.392156 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9"] Nov 03 17:15:00.400123 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.400094 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl"] Nov 03 17:15:00.404383 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.404357 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj"] Nov 03 17:15:00.410452 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.410428 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" Nov 03 17:15:00.426819 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.426783 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj"] Nov 03 17:15:00.432871 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.432832 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7"] Nov 03 17:15:00.439179 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.439151 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" Nov 03 17:15:00.449382 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.449240 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7"] Nov 03 17:15:00.474154 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.474112 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55e20055-4a9c-4d8b-8d33-3d7583e3f469-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55e20055-4a9c-4d8b-8d33-3d7583e3f469-plugins-config-volume\") pod \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\" (UID: \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\") " Nov 03 17:15:00.474322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.474173 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55e20055-4a9c-4d8b-8d33-3d7583e3f469-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55e20055-4a9c-4d8b-8d33-3d7583e3f469-routing-overlay-volume\") pod \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\" (UID: \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\") " Nov 03 17:15:00.474490 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.474467 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e20055-4a9c-4d8b-8d33-3d7583e3f469-provider-credentials\" (UniqueName: \"kubernetes.io/projected/55e20055-4a9c-4d8b-8d33-3d7583e3f469-provider-credentials\") pod \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\" (UID: \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\") " Nov 03 17:15:00.474534 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.474508 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e20055-4a9c-4d8b-8d33-3d7583e3f469-plugins-config-volume" pod "55e20055-4a9c-4d8b-8d33-3d7583e3f469" (UID: "55e20055-4a9c-4d8b-8d33-3d7583e3f469"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:00.474572 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.474534 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e20055-4a9c-4d8b-8d33-3d7583e3f469-kube-api-access-5hsww\" (UniqueName: \"kubernetes.io/projected/55e20055-4a9c-4d8b-8d33-3d7583e3f469-kube-api-access-5hsww\") pod \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\" (UID: \"55e20055-4a9c-4d8b-8d33-3d7583e3f469\") " Nov 03 17:15:00.474605 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.474576 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55e20055-4a9c-4d8b-8d33-3d7583e3f469-routing-overlay-volume" pod "55e20055-4a9c-4d8b-8d33-3d7583e3f469" (UID: "55e20055-4a9c-4d8b-8d33-3d7583e3f469"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:00.474930 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.474905 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5122b48b-0b08-4047-9245-a9c23e6a7187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b\" (UID: \"5122b48b-0b08-4047-9245-a9c23e6a7187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" Nov 03 17:15:00.475094 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.474977 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz4dr\" (UniqueName: \"kubernetes.io/projected/705f3454-7a40-4665-9d0d-0beec2f25e31-kube-api-access-hz4dr\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk\" (UID: \"705f3454-7a40-4665-9d0d-0beec2f25e31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" Nov 03 17:15:00.475094 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.475044 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkbhm\" (UniqueName: \"kubernetes.io/projected/32f33b7a-8400-4518-b408-6dd563d84820-kube-api-access-mkbhm\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9\" (UID: \"32f33b7a-8400-4518-b408-6dd563d84820\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" Nov 03 17:15:00.475326 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.475307 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32f33b7a-8400-4518-b408-6dd563d84820-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9\" (UID: \"32f33b7a-8400-4518-b408-6dd563d84820\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" Nov 03 17:15:00.475446 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.475362 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj\" (UID: \"2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" Nov 03 17:15:00.475516 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.475488 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djs4j\" (UniqueName: \"kubernetes.io/projected/5122b48b-0b08-4047-9245-a9c23e6a7187-kube-api-access-djs4j\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b\" (UID: \"5122b48b-0b08-4047-9245-a9c23e6a7187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" Nov 03 17:15:00.475579 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.475554 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/705f3454-7a40-4665-9d0d-0beec2f25e31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk\" (UID: \"705f3454-7a40-4665-9d0d-0beec2f25e31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" Nov 03 17:15:00.475749 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.475676 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtkfd\" (UniqueName: \"kubernetes.io/projected/6ac6d29c-ce79-4cd7-9dac-07836567e67f-kube-api-access-gtkfd\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr\" (UID: \"6ac6d29c-ce79-4cd7-9dac-07836567e67f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" Nov 03 17:15:00.475871 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.475834 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxkhx\" (UniqueName: \"kubernetes.io/projected/8afe859d-ebfc-4d31-9697-83198266d84a-kube-api-access-lxkhx\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl\" (UID: \"8afe859d-ebfc-4d31-9697-83198266d84a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" Nov 03 17:15:00.475961 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.475942 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkncs\" (UniqueName: \"kubernetes.io/projected/2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd-kube-api-access-xkncs\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj\" (UID: \"2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" Nov 03 17:15:00.476028 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.475990 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afe859d-ebfc-4d31-9697-83198266d84a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl\" (UID: \"8afe859d-ebfc-4d31-9697-83198266d84a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" Nov 03 17:15:00.476028 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.476022 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac6d29c-ce79-4cd7-9dac-07836567e67f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr\" (UID: \"6ac6d29c-ce79-4cd7-9dac-07836567e67f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" Nov 03 17:15:00.476283 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.476261 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55e20055-4a9c-4d8b-8d33-3d7583e3f469-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:00.476497 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.476289 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/55e20055-4a9c-4d8b-8d33-3d7583e3f469-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:00.477085 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.477042 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e20055-4a9c-4d8b-8d33-3d7583e3f469-provider-credentials" pod "55e20055-4a9c-4d8b-8d33-3d7583e3f469" (UID: "55e20055-4a9c-4d8b-8d33-3d7583e3f469"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:00.477453 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.477435 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e20055-4a9c-4d8b-8d33-3d7583e3f469-kube-api-access-5hsww" pod "55e20055-4a9c-4d8b-8d33-3d7583e3f469" (UID: "55e20055-4a9c-4d8b-8d33-3d7583e3f469"). InnerVolumeSpecName "kube-api-access-5hsww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:00.543184 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.543071 4947 generic.go:356] "Generic (PLEG): container finished" podID="55e20055-4a9c-4d8b-8d33-3d7583e3f469" containerID="c82788577110901a05e31c3d941cbffd5c77536bb5e996560707496fd1f51400" exitCode=0 Nov 03 17:15:00.543184 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.543143 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" Nov 03 17:15:00.543184 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.543154 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" event={"ID":"55e20055-4a9c-4d8b-8d33-3d7583e3f469","Type":"ContainerDied","Data":"c82788577110901a05e31c3d941cbffd5c77536bb5e996560707496fd1f51400"} Nov 03 17:15:00.543498 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.543219 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx" event={"ID":"55e20055-4a9c-4d8b-8d33-3d7583e3f469","Type":"ContainerDied","Data":"5022392eb13cbccd07140ec5a74740dd6928c3670f799497d005b07c7d5095e9"} Nov 03 17:15:00.543498 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.543243 4947 scope.go:122] "RemoveContainer" containerID="c82788577110901a05e31c3d941cbffd5c77536bb5e996560707496fd1f51400" Nov 03 17:15:00.556000 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.555976 4947 scope.go:122] "RemoveContainer" containerID="c82788577110901a05e31c3d941cbffd5c77536bb5e996560707496fd1f51400" Nov 03 17:15:00.556314 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:00.556293 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c82788577110901a05e31c3d941cbffd5c77536bb5e996560707496fd1f51400\": container with ID starting with c82788577110901a05e31c3d941cbffd5c77536bb5e996560707496fd1f51400 not found: ID does not exist" containerID="c82788577110901a05e31c3d941cbffd5c77536bb5e996560707496fd1f51400" Nov 03 17:15:00.556367 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.556326 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c82788577110901a05e31c3d941cbffd5c77536bb5e996560707496fd1f51400"} err="failed to get container status \"c82788577110901a05e31c3d941cbffd5c77536bb5e996560707496fd1f51400\": rpc error: code = NotFound desc = could not find container \"c82788577110901a05e31c3d941cbffd5c77536bb5e996560707496fd1f51400\": container with ID starting with c82788577110901a05e31c3d941cbffd5c77536bb5e996560707496fd1f51400 not found: ID does not exist" Nov 03 17:15:00.577635 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.577602 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5122b48b-0b08-4047-9245-a9c23e6a7187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b\" (UID: \"5122b48b-0b08-4047-9245-a9c23e6a7187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" Nov 03 17:15:00.577833 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.577640 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz4dr\" (UniqueName: \"kubernetes.io/projected/705f3454-7a40-4665-9d0d-0beec2f25e31-kube-api-access-hz4dr\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk\" (UID: \"705f3454-7a40-4665-9d0d-0beec2f25e31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" Nov 03 17:15:00.577833 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.577666 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkbhm\" (UniqueName: \"kubernetes.io/projected/32f33b7a-8400-4518-b408-6dd563d84820-kube-api-access-mkbhm\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9\" (UID: \"32f33b7a-8400-4518-b408-6dd563d84820\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" Nov 03 17:15:00.577833 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.577727 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32f33b7a-8400-4518-b408-6dd563d84820-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9\" (UID: \"32f33b7a-8400-4518-b408-6dd563d84820\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" Nov 03 17:15:00.577833 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.577769 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj\" (UID: \"2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" Nov 03 17:15:00.577833 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.577806 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djs4j\" (UniqueName: \"kubernetes.io/projected/5122b48b-0b08-4047-9245-a9c23e6a7187-kube-api-access-djs4j\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b\" (UID: \"5122b48b-0b08-4047-9245-a9c23e6a7187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" Nov 03 17:15:00.578119 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.577859 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/705f3454-7a40-4665-9d0d-0beec2f25e31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk\" (UID: \"705f3454-7a40-4665-9d0d-0beec2f25e31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" Nov 03 17:15:00.578119 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.577910 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtkfd\" (UniqueName: \"kubernetes.io/projected/6ac6d29c-ce79-4cd7-9dac-07836567e67f-kube-api-access-gtkfd\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr\" (UID: \"6ac6d29c-ce79-4cd7-9dac-07836567e67f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" Nov 03 17:15:00.578119 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.577987 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41406716-28a0-4b76-aeb5-c2a0a9b6292e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7\" (UID: \"41406716-28a0-4b76-aeb5-c2a0a9b6292e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" Nov 03 17:15:00.578119 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.578034 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxkhx\" (UniqueName: \"kubernetes.io/projected/8afe859d-ebfc-4d31-9697-83198266d84a-kube-api-access-lxkhx\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl\" (UID: \"8afe859d-ebfc-4d31-9697-83198266d84a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" Nov 03 17:15:00.578119 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.578115 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkncs\" (UniqueName: \"kubernetes.io/projected/2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd-kube-api-access-xkncs\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj\" (UID: \"2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" Nov 03 17:15:00.578388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.578169 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afe859d-ebfc-4d31-9697-83198266d84a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl\" (UID: \"8afe859d-ebfc-4d31-9697-83198266d84a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" Nov 03 17:15:00.578388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.578211 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac6d29c-ce79-4cd7-9dac-07836567e67f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr\" (UID: \"6ac6d29c-ce79-4cd7-9dac-07836567e67f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" Nov 03 17:15:00.578388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.578314 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfbdc\" (UniqueName: \"kubernetes.io/projected/41406716-28a0-4b76-aeb5-c2a0a9b6292e-kube-api-access-kfbdc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7\" (UID: \"41406716-28a0-4b76-aeb5-c2a0a9b6292e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" Nov 03 17:15:00.578557 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.578506 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32f33b7a-8400-4518-b408-6dd563d84820-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9\" (UID: \"32f33b7a-8400-4518-b408-6dd563d84820\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" Nov 03 17:15:00.578557 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.578523 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5122b48b-0b08-4047-9245-a9c23e6a7187-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b\" (UID: \"5122b48b-0b08-4047-9245-a9c23e6a7187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" Nov 03 17:15:00.578667 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.578636 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj\" (UID: \"2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" Nov 03 17:15:00.578667 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.578651 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/705f3454-7a40-4665-9d0d-0beec2f25e31-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk\" (UID: \"705f3454-7a40-4665-9d0d-0beec2f25e31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" Nov 03 17:15:00.578793 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.578681 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/55e20055-4a9c-4d8b-8d33-3d7583e3f469-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:00.578793 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.578755 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hsww\" (UniqueName: \"kubernetes.io/projected/55e20055-4a9c-4d8b-8d33-3d7583e3f469-kube-api-access-5hsww\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:00.579162 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.579146 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac6d29c-ce79-4cd7-9dac-07836567e67f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr\" (UID: \"6ac6d29c-ce79-4cd7-9dac-07836567e67f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" Nov 03 17:15:00.579215 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.579159 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afe859d-ebfc-4d31-9697-83198266d84a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl\" (UID: \"8afe859d-ebfc-4d31-9697-83198266d84a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" Nov 03 17:15:00.608335 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.608271 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkbhm\" (UniqueName: \"kubernetes.io/projected/32f33b7a-8400-4518-b408-6dd563d84820-kube-api-access-mkbhm\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9\" (UID: \"32f33b7a-8400-4518-b408-6dd563d84820\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" Nov 03 17:15:00.608335 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.608320 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkncs\" (UniqueName: \"kubernetes.io/projected/2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd-kube-api-access-xkncs\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj\" (UID: \"2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" Nov 03 17:15:00.608775 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.608749 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djs4j\" (UniqueName: \"kubernetes.io/projected/5122b48b-0b08-4047-9245-a9c23e6a7187-kube-api-access-djs4j\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b\" (UID: \"5122b48b-0b08-4047-9245-a9c23e6a7187\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" Nov 03 17:15:00.608835 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.608782 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz4dr\" (UniqueName: \"kubernetes.io/projected/705f3454-7a40-4665-9d0d-0beec2f25e31-kube-api-access-hz4dr\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk\" (UID: \"705f3454-7a40-4665-9d0d-0beec2f25e31\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" Nov 03 17:15:00.609312 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.609293 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxkhx\" (UniqueName: \"kubernetes.io/projected/8afe859d-ebfc-4d31-9697-83198266d84a-kube-api-access-lxkhx\") pod \"maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl\" (UID: \"8afe859d-ebfc-4d31-9697-83198266d84a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" Nov 03 17:15:00.609362 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.609326 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtkfd\" (UniqueName: \"kubernetes.io/projected/6ac6d29c-ce79-4cd7-9dac-07836567e67f-kube-api-access-gtkfd\") pod \"maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr\" (UID: \"6ac6d29c-ce79-4cd7-9dac-07836567e67f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" Nov 03 17:15:00.655012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.654975 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" Nov 03 17:15:00.669901 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.669588 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" Nov 03 17:15:00.683827 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.681077 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41406716-28a0-4b76-aeb5-c2a0a9b6292e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7\" (UID: \"41406716-28a0-4b76-aeb5-c2a0a9b6292e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" Nov 03 17:15:00.683827 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.681291 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfbdc\" (UniqueName: \"kubernetes.io/projected/41406716-28a0-4b76-aeb5-c2a0a9b6292e-kube-api-access-kfbdc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7\" (UID: \"41406716-28a0-4b76-aeb5-c2a0a9b6292e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" Nov 03 17:15:00.683827 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.682922 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41406716-28a0-4b76-aeb5-c2a0a9b6292e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7\" (UID: \"41406716-28a0-4b76-aeb5-c2a0a9b6292e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" Nov 03 17:15:00.701417 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.701382 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" Nov 03 17:15:00.712091 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.712058 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfbdc\" (UniqueName: \"kubernetes.io/projected/41406716-28a0-4b76-aeb5-c2a0a9b6292e-kube-api-access-kfbdc\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7\" (UID: \"41406716-28a0-4b76-aeb5-c2a0a9b6292e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" Nov 03 17:15:00.733503 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.733456 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" Nov 03 17:15:00.751021 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.750985 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" Nov 03 17:15:00.751274 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.751210 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx"] Nov 03 17:15:00.758930 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.757872 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3a0e27/payload-processing-external-model-e2e-extmod-a-3a0e27-74d9n9scx"] Nov 03 17:15:00.775510 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.775388 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" Nov 03 17:15:00.775510 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.775449 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" Nov 03 17:15:00.983101 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:00.983046 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b"] Nov 03 17:15:01.115573 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.115544 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9"] Nov 03 17:15:01.127206 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.127175 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7"] Nov 03 17:15:01.154315 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.154282 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl"] Nov 03 17:15:01.179107 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.179074 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj"] Nov 03 17:15:01.201472 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:15:01.201435 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f33b7a_8400_4518_b408_6dd563d84820.slice/crio-99c302b33c81d99e761dd66ebbff671aaac01bbb668568e692e0acf33f754160 WatchSource:0}: Error finding container 99c302b33c81d99e761dd66ebbff671aaac01bbb668568e692e0acf33f754160: Status 404 returned error can't find the container with id 99c302b33c81d99e761dd66ebbff671aaac01bbb668568e692e0acf33f754160 Nov 03 17:15:01.202328 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:15:01.202307 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41406716_28a0_4b76_aeb5_c2a0a9b6292e.slice/crio-2addcc855f328a48a4ab75ce6199f04ae0da13d5614ae470f1bd85ae873383ee WatchSource:0}: Error finding container 2addcc855f328a48a4ab75ce6199f04ae0da13d5614ae470f1bd85ae873383ee: Status 404 returned error can't find the container with id 2addcc855f328a48a4ab75ce6199f04ae0da13d5614ae470f1bd85ae873383ee Nov 03 17:15:01.211546 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.211524 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk"] Nov 03 17:15:01.212609 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:15:01.212582 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8a7026_f9bd_4ded_abcb_1e3c8ab733fd.slice/crio-efc572e653260146c87f86b71ada51b264c45163e96e09537660b9b2f6bbe1a1 WatchSource:0}: Error finding container efc572e653260146c87f86b71ada51b264c45163e96e09537660b9b2f6bbe1a1: Status 404 returned error can't find the container with id efc572e653260146c87f86b71ada51b264c45163e96e09537660b9b2f6bbe1a1 Nov 03 17:15:01.212830 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:15:01.212813 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705f3454_7a40_4665_9d0d_0beec2f25e31.slice/crio-39ec9b2cbd8deb6a6a99f01217e6226d3a10975d36f56cb53b95b8c07e5e4043 WatchSource:0}: Error finding container 39ec9b2cbd8deb6a6a99f01217e6226d3a10975d36f56cb53b95b8c07e5e4043: Status 404 returned error can't find the container with id 39ec9b2cbd8deb6a6a99f01217e6226d3a10975d36f56cb53b95b8c07e5e4043 Nov 03 17:15:01.271012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.270586 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr"] Nov 03 17:15:01.272068 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:15:01.272041 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac6d29c_ce79_4cd7_9dac_07836567e67f.slice/crio-a8ddbe4c15c7f4853c35c1cd12bf9a19f39853ec0cb142e2ea26702b17f7df30 WatchSource:0}: Error finding container a8ddbe4c15c7f4853c35c1cd12bf9a19f39853ec0cb142e2ea26702b17f7df30: Status 404 returned error can't find the container with id a8ddbe4c15c7f4853c35c1cd12bf9a19f39853ec0cb142e2ea26702b17f7df30 Nov 03 17:15:01.330785 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.330739 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" podUID="64232765-b0cc-498c-9041-048ed8285ad9" containerName="payload-processing" containerID="cri-o://8fcde897a2b8a0a65b2a9d6142c1def2a874eed399e9e2c761d47b775dad91b7" gracePeriod=25 Nov 03 17:15:01.557008 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.556979 4947 generic.go:356] "Generic (PLEG): container finished" podID="64232765-b0cc-498c-9041-048ed8285ad9" containerID="8fcde897a2b8a0a65b2a9d6142c1def2a874eed399e9e2c761d47b775dad91b7" exitCode=0 Nov 03 17:15:01.557115 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.557075 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" event={"ID":"64232765-b0cc-498c-9041-048ed8285ad9","Type":"ContainerDied","Data":"8fcde897a2b8a0a65b2a9d6142c1def2a874eed399e9e2c761d47b775dad91b7"} Nov 03 17:15:01.558482 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.558449 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" event={"ID":"8afe859d-ebfc-4d31-9697-83198266d84a","Type":"ContainerStarted","Data":"f84ff9c91a83fe1baac55c6d819819f2dc648be0995df8cb38c48213dbac0022"} Nov 03 17:15:01.562334 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.562306 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" event={"ID":"6ac6d29c-ce79-4cd7-9dac-07836567e67f","Type":"ContainerStarted","Data":"a8ddbe4c15c7f4853c35c1cd12bf9a19f39853ec0cb142e2ea26702b17f7df30"} Nov 03 17:15:01.563934 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.563911 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" event={"ID":"705f3454-7a40-4665-9d0d-0beec2f25e31","Type":"ContainerStarted","Data":"39ec9b2cbd8deb6a6a99f01217e6226d3a10975d36f56cb53b95b8c07e5e4043"} Nov 03 17:15:01.565351 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.565308 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" event={"ID":"41406716-28a0-4b76-aeb5-c2a0a9b6292e","Type":"ContainerStarted","Data":"2addcc855f328a48a4ab75ce6199f04ae0da13d5614ae470f1bd85ae873383ee"} Nov 03 17:15:01.569445 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.569422 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" event={"ID":"32f33b7a-8400-4518-b408-6dd563d84820","Type":"ContainerStarted","Data":"99c302b33c81d99e761dd66ebbff671aaac01bbb668568e692e0acf33f754160"} Nov 03 17:15:01.573399 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.572110 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" event={"ID":"5122b48b-0b08-4047-9245-a9c23e6a7187","Type":"ContainerStarted","Data":"20d567d727a2d5b60a41db4b3ccfcf68f06bc6e7de39e15fe59d4bed0cd1c93a"} Nov 03 17:15:01.573399 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.572159 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" event={"ID":"5122b48b-0b08-4047-9245-a9c23e6a7187","Type":"ContainerStarted","Data":"59c1af1c9074df45bf20ff6dab45d3a849a13149761bebb743df6b0ad0449a0d"} Nov 03 17:15:01.577487 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.577464 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" event={"ID":"2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd","Type":"ContainerStarted","Data":"efc572e653260146c87f86b71ada51b264c45163e96e09537660b9b2f6bbe1a1"} Nov 03 17:15:01.622672 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.622606 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" podStartSLOduration=1.6225865179999999 podStartE2EDuration="1.622586518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:01.616876845 +0000 UTC m=+3645.205876812" watchObservedRunningTime="2026-11-03 17:15:01.622586518 +0000 UTC m=+3645.211586666" Nov 03 17:15:01.720066 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.720010 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" Nov 03 17:15:01.807419 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.807392 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64232765-b0cc-498c-9041-048ed8285ad9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/64232765-b0cc-498c-9041-048ed8285ad9-provider-credentials\") pod \"64232765-b0cc-498c-9041-048ed8285ad9\" (UID: \"64232765-b0cc-498c-9041-048ed8285ad9\") " Nov 03 17:15:01.807597 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.807500 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64232765-b0cc-498c-9041-048ed8285ad9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64232765-b0cc-498c-9041-048ed8285ad9-routing-overlay-volume\") pod \"64232765-b0cc-498c-9041-048ed8285ad9\" (UID: \"64232765-b0cc-498c-9041-048ed8285ad9\") " Nov 03 17:15:01.807597 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.807561 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64232765-b0cc-498c-9041-048ed8285ad9-kube-api-access-kxtrg\" (UniqueName: \"kubernetes.io/projected/64232765-b0cc-498c-9041-048ed8285ad9-kube-api-access-kxtrg\") pod \"64232765-b0cc-498c-9041-048ed8285ad9\" (UID: \"64232765-b0cc-498c-9041-048ed8285ad9\") " Nov 03 17:15:01.807779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.807626 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64232765-b0cc-498c-9041-048ed8285ad9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64232765-b0cc-498c-9041-048ed8285ad9-plugins-config-volume\") pod \"64232765-b0cc-498c-9041-048ed8285ad9\" (UID: \"64232765-b0cc-498c-9041-048ed8285ad9\") " Nov 03 17:15:01.808688 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.808515 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64232765-b0cc-498c-9041-048ed8285ad9-plugins-config-volume" pod "64232765-b0cc-498c-9041-048ed8285ad9" (UID: "64232765-b0cc-498c-9041-048ed8285ad9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:01.809208 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.809184 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64232765-b0cc-498c-9041-048ed8285ad9-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:01.809344 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.809319 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64232765-b0cc-498c-9041-048ed8285ad9-routing-overlay-volume" pod "64232765-b0cc-498c-9041-048ed8285ad9" (UID: "64232765-b0cc-498c-9041-048ed8285ad9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:01.813183 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.813152 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64232765-b0cc-498c-9041-048ed8285ad9-provider-credentials" pod "64232765-b0cc-498c-9041-048ed8285ad9" (UID: "64232765-b0cc-498c-9041-048ed8285ad9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:01.814728 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.814674 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64232765-b0cc-498c-9041-048ed8285ad9-kube-api-access-kxtrg" pod "64232765-b0cc-498c-9041-048ed8285ad9" (UID: "64232765-b0cc-498c-9041-048ed8285ad9"). InnerVolumeSpecName "kube-api-access-kxtrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:01.911502 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.911476 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64232765-b0cc-498c-9041-048ed8285ad9-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:01.911580 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.911505 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64232765-b0cc-498c-9041-048ed8285ad9-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:01.911580 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:01.911515 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxtrg\" (UniqueName: \"kubernetes.io/projected/64232765-b0cc-498c-9041-048ed8285ad9-kube-api-access-kxtrg\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:02.568046 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.567975 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55e20055-4a9c-4d8b-8d33-3d7583e3f469" path="/var/lib/kubelet/pods/55e20055-4a9c-4d8b-8d33-3d7583e3f469/volumes" Nov 03 17:15:02.586559 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.586525 4947 generic.go:356] "Generic (PLEG): container finished" podID="6ac6d29c-ce79-4cd7-9dac-07836567e67f" containerID="2e9e765c51ddcfe416efcedcf6696209b1e363c85ef455015166076abaf09173" exitCode=60 Nov 03 17:15:02.586744 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.586611 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" event={"ID":"6ac6d29c-ce79-4cd7-9dac-07836567e67f","Type":"ContainerDied","Data":"2e9e765c51ddcfe416efcedcf6696209b1e363c85ef455015166076abaf09173"} Nov 03 17:15:02.586966 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.586947 4947 scope.go:122] "RemoveContainer" containerID="2e9e765c51ddcfe416efcedcf6696209b1e363c85ef455015166076abaf09173" Nov 03 17:15:02.588473 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.588445 4947 generic.go:356] "Generic (PLEG): container finished" podID="705f3454-7a40-4665-9d0d-0beec2f25e31" containerID="7442fbc98f29d034e4a64beb99783e5688b91bfaff76db7a7809ddae483654c8" exitCode=60 Nov 03 17:15:02.588576 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.588538 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" event={"ID":"705f3454-7a40-4665-9d0d-0beec2f25e31","Type":"ContainerDied","Data":"7442fbc98f29d034e4a64beb99783e5688b91bfaff76db7a7809ddae483654c8"} Nov 03 17:15:02.588875 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.588807 4947 scope.go:122] "RemoveContainer" containerID="7442fbc98f29d034e4a64beb99783e5688b91bfaff76db7a7809ddae483654c8" Nov 03 17:15:02.590500 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.590474 4947 generic.go:356] "Generic (PLEG): container finished" podID="41406716-28a0-4b76-aeb5-c2a0a9b6292e" containerID="2ba2bd3a96ca68dfb1e5ece4a3509f5f7890d5807527d34859c346a859c606ec" exitCode=60 Nov 03 17:15:02.590673 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.590596 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" event={"ID":"41406716-28a0-4b76-aeb5-c2a0a9b6292e","Type":"ContainerDied","Data":"2ba2bd3a96ca68dfb1e5ece4a3509f5f7890d5807527d34859c346a859c606ec"} Nov 03 17:15:02.590912 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.590875 4947 scope.go:122] "RemoveContainer" containerID="2ba2bd3a96ca68dfb1e5ece4a3509f5f7890d5807527d34859c346a859c606ec" Nov 03 17:15:02.592570 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.592504 4947 generic.go:356] "Generic (PLEG): container finished" podID="32f33b7a-8400-4518-b408-6dd563d84820" containerID="76f6d78d42cea6f56b4509e16fb18df4c564791860afd2ab6c5f230822f35a0e" exitCode=60 Nov 03 17:15:02.592634 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.592603 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" event={"ID":"32f33b7a-8400-4518-b408-6dd563d84820","Type":"ContainerDied","Data":"76f6d78d42cea6f56b4509e16fb18df4c564791860afd2ab6c5f230822f35a0e"} Nov 03 17:15:02.592861 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.592845 4947 scope.go:122] "RemoveContainer" containerID="76f6d78d42cea6f56b4509e16fb18df4c564791860afd2ab6c5f230822f35a0e" Nov 03 17:15:02.594126 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.594106 4947 generic.go:356] "Generic (PLEG): container finished" podID="5122b48b-0b08-4047-9245-a9c23e6a7187" containerID="20d567d727a2d5b60a41db4b3ccfcf68f06bc6e7de39e15fe59d4bed0cd1c93a" exitCode=60 Nov 03 17:15:02.594213 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.594128 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" event={"ID":"5122b48b-0b08-4047-9245-a9c23e6a7187","Type":"ContainerDied","Data":"20d567d727a2d5b60a41db4b3ccfcf68f06bc6e7de39e15fe59d4bed0cd1c93a"} Nov 03 17:15:02.594416 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.594399 4947 scope.go:122] "RemoveContainer" containerID="20d567d727a2d5b60a41db4b3ccfcf68f06bc6e7de39e15fe59d4bed0cd1c93a" Nov 03 17:15:02.595619 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.595601 4947 generic.go:356] "Generic (PLEG): container finished" podID="2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd" containerID="a15ab1cc002edcc5eda9ebe439636856b37a4620e595d8f0a53b4ce933b58bca" exitCode=60 Nov 03 17:15:02.595747 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.595676 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" event={"ID":"2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd","Type":"ContainerDied","Data":"a15ab1cc002edcc5eda9ebe439636856b37a4620e595d8f0a53b4ce933b58bca"} Nov 03 17:15:02.595922 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.595905 4947 scope.go:122] "RemoveContainer" containerID="a15ab1cc002edcc5eda9ebe439636856b37a4620e595d8f0a53b4ce933b58bca" Nov 03 17:15:02.597056 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.597041 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" Nov 03 17:15:02.597056 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.597046 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd" event={"ID":"64232765-b0cc-498c-9041-048ed8285ad9","Type":"ContainerDied","Data":"59b9f953720b3c4cc293f372ec85a8aa7c93e631900c33c3d01943ad7f5b5da4"} Nov 03 17:15:02.597202 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.597079 4947 scope.go:122] "RemoveContainer" containerID="8fcde897a2b8a0a65b2a9d6142c1def2a874eed399e9e2c761d47b775dad91b7" Nov 03 17:15:02.599534 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.599371 4947 generic.go:356] "Generic (PLEG): container finished" podID="8afe859d-ebfc-4d31-9697-83198266d84a" containerID="b020fd0b35b406bb12c913097f18f027039c4baf9311a43da476dfbaf2334329" exitCode=60 Nov 03 17:15:02.599534 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.599493 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" event={"ID":"8afe859d-ebfc-4d31-9697-83198266d84a","Type":"ContainerDied","Data":"b020fd0b35b406bb12c913097f18f027039c4baf9311a43da476dfbaf2334329"} Nov 03 17:15:02.599900 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.599879 4947 scope.go:122] "RemoveContainer" containerID="b020fd0b35b406bb12c913097f18f027039c4baf9311a43da476dfbaf2334329" Nov 03 17:15:02.947468 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.947436 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd"] Nov 03 17:15:02.951980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:02.951953 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-790111/payload-processing-external-model-e2e-extmod-b-790111-55bflx4xd"] Nov 03 17:15:03.610981 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:03.610946 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" event={"ID":"6ac6d29c-ce79-4cd7-9dac-07836567e67f","Type":"ContainerStarted","Data":"40ba2b1375314074c6fee4130582e60d163d42c384dbc764dc33d1cf097945c1"} Nov 03 17:15:03.612996 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:03.612970 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" event={"ID":"705f3454-7a40-4665-9d0d-0beec2f25e31","Type":"ContainerStarted","Data":"a2062fb0c679ff393527a1fa61ebf1f359c3918866de934e46a2beeb374b92ab"} Nov 03 17:15:03.614757 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:03.614728 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" event={"ID":"41406716-28a0-4b76-aeb5-c2a0a9b6292e","Type":"ContainerStarted","Data":"693102659ddc0080d45e6c2caabae78e080faf46f657b2f7b66a77efb89369ae"} Nov 03 17:15:03.616432 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:03.616403 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" event={"ID":"32f33b7a-8400-4518-b408-6dd563d84820","Type":"ContainerStarted","Data":"3db300f8b2ce764de1af68753f1b293eaacb6f2333dd74a282afff61dd202c0a"} Nov 03 17:15:03.618080 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:03.618061 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" event={"ID":"5122b48b-0b08-4047-9245-a9c23e6a7187","Type":"ContainerStarted","Data":"26af6fbacf5bb37d68bf65e19c55f53d854bd3bc059b666396be689466bbb228"} Nov 03 17:15:03.619754 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:03.619730 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" event={"ID":"2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd","Type":"ContainerStarted","Data":"88bc1a67ee3173222e78dab6a51f0539babfda2e3f573e6e938ae9f27f8d2136"} Nov 03 17:15:03.622211 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:03.622188 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" event={"ID":"8afe859d-ebfc-4d31-9697-83198266d84a","Type":"ContainerStarted","Data":"906d88f01a6ca30ec07be91cf3a3cc4d89a8a9719c95e5d00dfe47b474c339af"} Nov 03 17:15:03.662473 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:03.662409 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" podStartSLOduration=3.66239474 podStartE2EDuration="3.66239474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:03.65738721 +0000 UTC m=+3647.246387178" watchObservedRunningTime="2026-11-03 17:15:03.66239474 +0000 UTC m=+3647.251394708" Nov 03 17:15:03.753627 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:03.753504 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" podStartSLOduration=3.7534836609999997 podStartE2EDuration="3.753483661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:03.746232449 +0000 UTC m=+3647.335232419" watchObservedRunningTime="2026-11-03 17:15:03.753483661 +0000 UTC m=+3647.342483706" Nov 03 17:15:03.793940 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:03.793877 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" podStartSLOduration=3.793856887 podStartE2EDuration="3.793856887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:03.789619943 +0000 UTC m=+3647.378619910" watchObservedRunningTime="2026-11-03 17:15:03.793856887 +0000 UTC m=+3647.382856856" Nov 03 17:15:03.833804 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:03.833752 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" podStartSLOduration=3.8337378749999997 podStartE2EDuration="3.833737875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:03.832955026 +0000 UTC m=+3647.421954994" watchObservedRunningTime="2026-11-03 17:15:03.833737875 +0000 UTC m=+3647.422737843" Nov 03 17:15:03.882274 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:03.882166 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" podStartSLOduration=3.88215026 podStartE2EDuration="3.88215026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:03.877391909 +0000 UTC m=+3647.466391928" watchObservedRunningTime="2026-11-03 17:15:03.88215026 +0000 UTC m=+3647.471150228" Nov 03 17:15:03.924732 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:03.924662 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" podStartSLOduration=3.924648247 podStartE2EDuration="3.924648247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:03.921000134 +0000 UTC m=+3647.510000101" watchObservedRunningTime="2026-11-03 17:15:03.924648247 +0000 UTC m=+3647.513648214" Nov 03 17:15:04.318147 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.318105 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct"] Nov 03 17:15:04.319503 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.319470 4947 cpu_manager.go:414] "containerMap: removing container" podUID="64232765-b0cc-498c-9041-048ed8285ad9" containerName="payload-processing" Nov 03 17:15:04.319503 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.319499 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64232765-b0cc-498c-9041-048ed8285ad9" containerName="payload-processing" Nov 03 17:15:04.319810 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.319792 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="64232765-b0cc-498c-9041-048ed8285ad9" containerName="payload-processing" Nov 03 17:15:04.324005 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.323984 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.350574 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.350540 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct"] Nov 03 17:15:04.351943 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.351907 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-cb0fa3-openshift-default-dockercfg-67frl\"" Nov 03 17:15:04.490777 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.485950 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-istio-data\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.490777 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.486014 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-workload-socket\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.490777 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.486120 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-credential-socket\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.490777 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.486150 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-istio-envoy\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.490777 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.486216 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wms95\" (UniqueName: \"kubernetes.io/projected/c7a45866-e8e8-4731-b688-4674bf874a44-kube-api-access-wms95\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.490777 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.486289 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a45866-e8e8-4731-b688-4674bf874a44-istio-podinfo\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.490777 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.486395 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a45866-e8e8-4731-b688-4674bf874a44-istio-token\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.490777 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.486454 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-workload-certs\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.490777 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.486639 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a45866-e8e8-4731-b688-4674bf874a44-istiod-ca-cert\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.590731 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.590112 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a45866-e8e8-4731-b688-4674bf874a44-istio-token\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.590731 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.590189 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-workload-certs\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.590731 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.590332 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a45866-e8e8-4731-b688-4674bf874a44-istiod-ca-cert\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.590731 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.590439 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-istio-data\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.590731 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.590477 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-workload-socket\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.590731 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.590559 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-credential-socket\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.590731 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.590586 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-istio-envoy\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.590731 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.590644 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wms95\" (UniqueName: \"kubernetes.io/projected/c7a45866-e8e8-4731-b688-4674bf874a44-kube-api-access-wms95\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.597742 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.595145 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-istio-data\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.614719 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.611034 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-workload-certs\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.614719 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.611995 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a45866-e8e8-4731-b688-4674bf874a44-istiod-ca-cert\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.614719 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.612054 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a45866-e8e8-4731-b688-4674bf874a44-istio-podinfo\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.614719 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.612271 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-workload-socket\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.614719 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.612556 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-credential-socket\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.676592 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.657269 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a45866-e8e8-4731-b688-4674bf874a44-istio-token\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.676592 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.665482 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a45866-e8e8-4731-b688-4674bf874a44-istio-podinfo\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.696691 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.696507 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wms95\" (UniqueName: \"kubernetes.io/projected/c7a45866-e8e8-4731-b688-4674bf874a44-kube-api-access-wms95\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.700366 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.700034 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-istio-envoy\") pod \"e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:04.701972 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.701360 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64232765-b0cc-498c-9041-048ed8285ad9" path="/var/lib/kubelet/pods/64232765-b0cc-498c-9041-048ed8285ad9/volumes" Nov 03 17:15:04.744108 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.741677 4947 generic.go:356] "Generic (PLEG): container finished" podID="6ac6d29c-ce79-4cd7-9dac-07836567e67f" containerID="40ba2b1375314074c6fee4130582e60d163d42c384dbc764dc33d1cf097945c1" exitCode=60 Nov 03 17:15:04.744108 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.741822 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" event={"ID":"6ac6d29c-ce79-4cd7-9dac-07836567e67f","Type":"ContainerDied","Data":"40ba2b1375314074c6fee4130582e60d163d42c384dbc764dc33d1cf097945c1"} Nov 03 17:15:04.744108 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.741887 4947 scope.go:122] "RemoveContainer" containerID="2e9e765c51ddcfe416efcedcf6696209b1e363c85ef455015166076abaf09173" Nov 03 17:15:04.744108 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.742452 4947 scope.go:122] "RemoveContainer" containerID="40ba2b1375314074c6fee4130582e60d163d42c384dbc764dc33d1cf097945c1" Nov 03 17:15:04.744108 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:04.742998 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr_odh-ai-gateway-infra(6ac6d29c-ce79-4cd7-9dac-07836567e67f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" podUID="6ac6d29c-ce79-4cd7-9dac-07836567e67f" Nov 03 17:15:04.751744 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.750227 4947 generic.go:356] "Generic (PLEG): container finished" podID="705f3454-7a40-4665-9d0d-0beec2f25e31" containerID="a2062fb0c679ff393527a1fa61ebf1f359c3918866de934e46a2beeb374b92ab" exitCode=60 Nov 03 17:15:04.751744 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.750325 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" event={"ID":"705f3454-7a40-4665-9d0d-0beec2f25e31","Type":"ContainerDied","Data":"a2062fb0c679ff393527a1fa61ebf1f359c3918866de934e46a2beeb374b92ab"} Nov 03 17:15:04.751744 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.750947 4947 scope.go:122] "RemoveContainer" containerID="a2062fb0c679ff393527a1fa61ebf1f359c3918866de934e46a2beeb374b92ab" Nov 03 17:15:04.753346 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:04.753293 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk_odh-ai-gateway-infra(705f3454-7a40-4665-9d0d-0beec2f25e31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" podUID="705f3454-7a40-4665-9d0d-0beec2f25e31" Nov 03 17:15:04.756659 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.756617 4947 generic.go:356] "Generic (PLEG): container finished" podID="41406716-28a0-4b76-aeb5-c2a0a9b6292e" containerID="693102659ddc0080d45e6c2caabae78e080faf46f657b2f7b66a77efb89369ae" exitCode=60 Nov 03 17:15:04.757057 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.756958 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" event={"ID":"41406716-28a0-4b76-aeb5-c2a0a9b6292e","Type":"ContainerDied","Data":"693102659ddc0080d45e6c2caabae78e080faf46f657b2f7b66a77efb89369ae"} Nov 03 17:15:04.771794 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.768346 4947 generic.go:356] "Generic (PLEG): container finished" podID="32f33b7a-8400-4518-b408-6dd563d84820" containerID="3db300f8b2ce764de1af68753f1b293eaacb6f2333dd74a282afff61dd202c0a" exitCode=60 Nov 03 17:15:04.771794 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.768512 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" event={"ID":"32f33b7a-8400-4518-b408-6dd563d84820","Type":"ContainerDied","Data":"3db300f8b2ce764de1af68753f1b293eaacb6f2333dd74a282afff61dd202c0a"} Nov 03 17:15:04.771794 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.769814 4947 scope.go:122] "RemoveContainer" containerID="693102659ddc0080d45e6c2caabae78e080faf46f657b2f7b66a77efb89369ae" Nov 03 17:15:04.771794 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:04.770413 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7_odh-ai-gateway-infra(41406716-28a0-4b76-aeb5-c2a0a9b6292e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" podUID="41406716-28a0-4b76-aeb5-c2a0a9b6292e" Nov 03 17:15:04.771794 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.770603 4947 scope.go:122] "RemoveContainer" containerID="3db300f8b2ce764de1af68753f1b293eaacb6f2333dd74a282afff61dd202c0a" Nov 03 17:15:04.771794 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:04.771058 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9_odh-ai-gateway-infra(32f33b7a-8400-4518-b408-6dd563d84820)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" podUID="32f33b7a-8400-4518-b408-6dd563d84820" Nov 03 17:15:04.797091 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.797052 4947 generic.go:356] "Generic (PLEG): container finished" podID="5122b48b-0b08-4047-9245-a9c23e6a7187" containerID="26af6fbacf5bb37d68bf65e19c55f53d854bd3bc059b666396be689466bbb228" exitCode=60 Nov 03 17:15:04.797421 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.797390 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" event={"ID":"5122b48b-0b08-4047-9245-a9c23e6a7187","Type":"ContainerDied","Data":"26af6fbacf5bb37d68bf65e19c55f53d854bd3bc059b666396be689466bbb228"} Nov 03 17:15:04.798309 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.797762 4947 scope.go:122] "RemoveContainer" containerID="26af6fbacf5bb37d68bf65e19c55f53d854bd3bc059b666396be689466bbb228" Nov 03 17:15:04.798500 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.798479 4947 scope.go:122] "RemoveContainer" containerID="7442fbc98f29d034e4a64beb99783e5688b91bfaff76db7a7809ddae483654c8" Nov 03 17:15:04.799174 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:04.799087 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b_odh-ai-gateway-infra(5122b48b-0b08-4047-9245-a9c23e6a7187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" podUID="5122b48b-0b08-4047-9245-a9c23e6a7187" Nov 03 17:15:04.802825 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.802799 4947 generic.go:356] "Generic (PLEG): container finished" podID="2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd" containerID="88bc1a67ee3173222e78dab6a51f0539babfda2e3f573e6e938ae9f27f8d2136" exitCode=60 Nov 03 17:15:04.802950 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.802872 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" event={"ID":"2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd","Type":"ContainerDied","Data":"88bc1a67ee3173222e78dab6a51f0539babfda2e3f573e6e938ae9f27f8d2136"} Nov 03 17:15:04.803271 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.803249 4947 scope.go:122] "RemoveContainer" containerID="88bc1a67ee3173222e78dab6a51f0539babfda2e3f573e6e938ae9f27f8d2136" Nov 03 17:15:04.803648 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:04.803624 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj_odh-ai-gateway-infra(2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" podUID="2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd" Nov 03 17:15:04.805452 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.805426 4947 generic.go:356] "Generic (PLEG): container finished" podID="8afe859d-ebfc-4d31-9697-83198266d84a" containerID="906d88f01a6ca30ec07be91cf3a3cc4d89a8a9719c95e5d00dfe47b474c339af" exitCode=60 Nov 03 17:15:04.805595 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.805502 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" event={"ID":"8afe859d-ebfc-4d31-9697-83198266d84a","Type":"ContainerDied","Data":"906d88f01a6ca30ec07be91cf3a3cc4d89a8a9719c95e5d00dfe47b474c339af"} Nov 03 17:15:04.805882 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.805859 4947 scope.go:122] "RemoveContainer" containerID="906d88f01a6ca30ec07be91cf3a3cc4d89a8a9719c95e5d00dfe47b474c339af" Nov 03 17:15:04.806222 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:04.806203 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl_odh-ai-gateway-infra(8afe859d-ebfc-4d31-9697-83198266d84a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" podUID="8afe859d-ebfc-4d31-9697-83198266d84a" Nov 03 17:15:04.826354 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.826319 4947 scope.go:122] "RemoveContainer" containerID="2ba2bd3a96ca68dfb1e5ece4a3509f5f7890d5807527d34859c346a859c606ec" Nov 03 17:15:04.844246 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.844219 4947 scope.go:122] "RemoveContainer" containerID="76f6d78d42cea6f56b4509e16fb18df4c564791860afd2ab6c5f230822f35a0e" Nov 03 17:15:04.862618 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.862514 4947 scope.go:122] "RemoveContainer" containerID="20d567d727a2d5b60a41db4b3ccfcf68f06bc6e7de39e15fe59d4bed0cd1c93a" Nov 03 17:15:04.878081 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.877988 4947 scope.go:122] "RemoveContainer" containerID="a15ab1cc002edcc5eda9ebe439636856b37a4620e595d8f0a53b4ce933b58bca" Nov 03 17:15:04.895097 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.895071 4947 scope.go:122] "RemoveContainer" containerID="b020fd0b35b406bb12c913097f18f027039c4baf9311a43da476dfbaf2334329" Nov 03 17:15:04.947418 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:04.947386 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:05.321628 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:15:05.321598 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a45866_e8e8_4731_b688_4674bf874a44.slice/crio-4ff6884ee1a6473b7bfeca7bb3df269dd7d359f406647c125e92473abed28015 WatchSource:0}: Error finding container 4ff6884ee1a6473b7bfeca7bb3df269dd7d359f406647c125e92473abed28015: Status 404 returned error can't find the container with id 4ff6884ee1a6473b7bfeca7bb3df269dd7d359f406647c125e92473abed28015 Nov 03 17:15:05.325093 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:05.325053 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:15:05.325200 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:05.325118 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct"] Nov 03 17:15:05.325200 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:05.325135 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:15:05.325200 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:05.325182 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:15:05.817661 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:05.817625 4947 scope.go:122] "RemoveContainer" containerID="693102659ddc0080d45e6c2caabae78e080faf46f657b2f7b66a77efb89369ae" Nov 03 17:15:05.818197 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:05.818131 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7_odh-ai-gateway-infra(41406716-28a0-4b76-aeb5-c2a0a9b6292e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" podUID="41406716-28a0-4b76-aeb5-c2a0a9b6292e" Nov 03 17:15:05.819599 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:05.819577 4947 scope.go:122] "RemoveContainer" containerID="3db300f8b2ce764de1af68753f1b293eaacb6f2333dd74a282afff61dd202c0a" Nov 03 17:15:05.820112 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:05.820083 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9_odh-ai-gateway-infra(32f33b7a-8400-4518-b408-6dd563d84820)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" podUID="32f33b7a-8400-4518-b408-6dd563d84820" Nov 03 17:15:05.821407 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:05.821384 4947 scope.go:122] "RemoveContainer" containerID="26af6fbacf5bb37d68bf65e19c55f53d854bd3bc059b666396be689466bbb228" Nov 03 17:15:05.821791 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:05.821774 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b_odh-ai-gateway-infra(5122b48b-0b08-4047-9245-a9c23e6a7187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" podUID="5122b48b-0b08-4047-9245-a9c23e6a7187" Nov 03 17:15:05.823157 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:05.823132 4947 scope.go:122] "RemoveContainer" containerID="88bc1a67ee3173222e78dab6a51f0539babfda2e3f573e6e938ae9f27f8d2136" Nov 03 17:15:05.823514 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:05.823492 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj_odh-ai-gateway-infra(2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" podUID="2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd" Nov 03 17:15:05.824844 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:05.824826 4947 scope.go:122] "RemoveContainer" containerID="906d88f01a6ca30ec07be91cf3a3cc4d89a8a9719c95e5d00dfe47b474c339af" Nov 03 17:15:05.825141 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:05.825120 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl_odh-ai-gateway-infra(8afe859d-ebfc-4d31-9697-83198266d84a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" podUID="8afe859d-ebfc-4d31-9697-83198266d84a" Nov 03 17:15:05.826266 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:05.826246 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" event={"ID":"c7a45866-e8e8-4731-b688-4674bf874a44","Type":"ContainerStarted","Data":"60ef2342f8d84e076a0b8bffa2ac96049cdd911af75d4f35580ee3e100d003aa"} Nov 03 17:15:05.826374 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:05.826298 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" event={"ID":"c7a45866-e8e8-4731-b688-4674bf874a44","Type":"ContainerStarted","Data":"4ff6884ee1a6473b7bfeca7bb3df269dd7d359f406647c125e92473abed28015"} Nov 03 17:15:05.828546 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:05.828531 4947 scope.go:122] "RemoveContainer" containerID="40ba2b1375314074c6fee4130582e60d163d42c384dbc764dc33d1cf097945c1" Nov 03 17:15:05.828840 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:05.828816 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr_odh-ai-gateway-infra(6ac6d29c-ce79-4cd7-9dac-07836567e67f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" podUID="6ac6d29c-ce79-4cd7-9dac-07836567e67f" Nov 03 17:15:05.830267 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:05.830251 4947 scope.go:122] "RemoveContainer" containerID="a2062fb0c679ff393527a1fa61ebf1f359c3918866de934e46a2beeb374b92ab" Nov 03 17:15:05.830484 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:05.830467 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk_odh-ai-gateway-infra(705f3454-7a40-4665-9d0d-0beec2f25e31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" podUID="705f3454-7a40-4665-9d0d-0beec2f25e31" Nov 03 17:15:05.948426 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:05.948394 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:06.050472 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:06.050416 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" podStartSLOduration=2.050401254 podStartE2EDuration="2.050401254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:06.048190606 +0000 UTC m=+3649.637190575" watchObservedRunningTime="2026-11-03 17:15:06.050401254 +0000 UTC m=+3649.639401247" Nov 03 17:15:06.955058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:06.955021 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:07.854468 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:07.854437 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:07.855633 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:07.855614 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:15:10.547422 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:10.547390 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" Nov 03 17:15:10.547884 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:10.547479 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" Nov 03 17:15:14.568181 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:14.568150 4947 scope.go:122] "RemoveContainer" containerID="3db300f8b2ce764de1af68753f1b293eaacb6f2333dd74a282afff61dd202c0a" Nov 03 17:15:15.558154 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:15.558072 4947 scope.go:122] "RemoveContainer" containerID="26af6fbacf5bb37d68bf65e19c55f53d854bd3bc059b666396be689466bbb228" Nov 03 17:15:16.031676 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:16.031644 4947 generic.go:356] "Generic (PLEG): container finished" podID="32f33b7a-8400-4518-b408-6dd563d84820" containerID="91211d79d7dfd6a6e6df1adbd463436917838e9f1e52da093c1861f746112488" exitCode=60 Nov 03 17:15:16.032177 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:16.031719 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" event={"ID":"32f33b7a-8400-4518-b408-6dd563d84820","Type":"ContainerDied","Data":"91211d79d7dfd6a6e6df1adbd463436917838e9f1e52da093c1861f746112488"} Nov 03 17:15:16.032177 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:16.031788 4947 scope.go:122] "RemoveContainer" containerID="3db300f8b2ce764de1af68753f1b293eaacb6f2333dd74a282afff61dd202c0a" Nov 03 17:15:16.032310 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:16.032254 4947 scope.go:122] "RemoveContainer" containerID="91211d79d7dfd6a6e6df1adbd463436917838e9f1e52da093c1861f746112488" Nov 03 17:15:16.032753 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:16.032725 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9_odh-ai-gateway-infra(32f33b7a-8400-4518-b408-6dd563d84820)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" podUID="32f33b7a-8400-4518-b408-6dd563d84820" Nov 03 17:15:16.034422 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:16.034394 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" event={"ID":"5122b48b-0b08-4047-9245-a9c23e6a7187","Type":"ContainerStarted","Data":"5e0adfc79180711e4bcb3d318378720d9fc1637299ca4da88c44cad6762bfe79"} Nov 03 17:15:16.569738 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:16.569710 4947 scope.go:122] "RemoveContainer" containerID="88bc1a67ee3173222e78dab6a51f0539babfda2e3f573e6e938ae9f27f8d2136" Nov 03 17:15:16.569930 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:16.569908 4947 scope.go:122] "RemoveContainer" containerID="906d88f01a6ca30ec07be91cf3a3cc4d89a8a9719c95e5d00dfe47b474c339af" Nov 03 17:15:16.569987 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:16.569963 4947 scope.go:122] "RemoveContainer" containerID="40ba2b1375314074c6fee4130582e60d163d42c384dbc764dc33d1cf097945c1" Nov 03 17:15:17.044345 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:17.044311 4947 generic.go:356] "Generic (PLEG): container finished" podID="5122b48b-0b08-4047-9245-a9c23e6a7187" containerID="5e0adfc79180711e4bcb3d318378720d9fc1637299ca4da88c44cad6762bfe79" exitCode=60 Nov 03 17:15:17.044897 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:17.044391 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" event={"ID":"5122b48b-0b08-4047-9245-a9c23e6a7187","Type":"ContainerDied","Data":"5e0adfc79180711e4bcb3d318378720d9fc1637299ca4da88c44cad6762bfe79"} Nov 03 17:15:17.044897 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:17.044461 4947 scope.go:122] "RemoveContainer" containerID="26af6fbacf5bb37d68bf65e19c55f53d854bd3bc059b666396be689466bbb228" Nov 03 17:15:17.045071 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:17.045053 4947 scope.go:122] "RemoveContainer" containerID="5e0adfc79180711e4bcb3d318378720d9fc1637299ca4da88c44cad6762bfe79" Nov 03 17:15:17.045638 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:17.045523 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b_odh-ai-gateway-infra(5122b48b-0b08-4047-9245-a9c23e6a7187)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" podUID="5122b48b-0b08-4047-9245-a9c23e6a7187" Nov 03 17:15:17.095097 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:17.095018 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9"] Nov 03 17:15:17.137345 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:17.137318 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b"] Nov 03 17:15:17.292499 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:17.292471 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" Nov 03 17:15:17.366925 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:17.366784 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32f33b7a-8400-4518-b408-6dd563d84820-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32f33b7a-8400-4518-b408-6dd563d84820-maas-api-service-ca\") pod \"32f33b7a-8400-4518-b408-6dd563d84820\" (UID: \"32f33b7a-8400-4518-b408-6dd563d84820\") " Nov 03 17:15:17.367231 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:17.367206 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f33b7a-8400-4518-b408-6dd563d84820-kube-api-access-mkbhm\" (UniqueName: \"kubernetes.io/projected/32f33b7a-8400-4518-b408-6dd563d84820-kube-api-access-mkbhm\") pod \"32f33b7a-8400-4518-b408-6dd563d84820\" (UID: \"32f33b7a-8400-4518-b408-6dd563d84820\") " Nov 03 17:15:17.370460 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:17.370423 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f33b7a-8400-4518-b408-6dd563d84820-maas-api-service-ca" pod "32f33b7a-8400-4518-b408-6dd563d84820" (UID: "32f33b7a-8400-4518-b408-6dd563d84820"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:17.372591 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:17.372559 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f33b7a-8400-4518-b408-6dd563d84820-kube-api-access-mkbhm" pod "32f33b7a-8400-4518-b408-6dd563d84820" (UID: "32f33b7a-8400-4518-b408-6dd563d84820"). InnerVolumeSpecName "kube-api-access-mkbhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:17.470765 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:17.470727 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32f33b7a-8400-4518-b408-6dd563d84820-maas-api-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:17.470765 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:17.470758 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkbhm\" (UniqueName: \"kubernetes.io/projected/32f33b7a-8400-4518-b408-6dd563d84820-kube-api-access-mkbhm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:17.558563 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:17.558528 4947 scope.go:122] "RemoveContainer" containerID="693102659ddc0080d45e6c2caabae78e080faf46f657b2f7b66a77efb89369ae" Nov 03 17:15:18.058690 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.058659 4947 generic.go:356] "Generic (PLEG): container finished" podID="2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd" containerID="def2b245503c7613091672b5c052d4e82d87b8f821152e81aa034295b8d63871" exitCode=60 Nov 03 17:15:18.059153 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.058752 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" event={"ID":"2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd","Type":"ContainerDied","Data":"def2b245503c7613091672b5c052d4e82d87b8f821152e81aa034295b8d63871"} Nov 03 17:15:18.059153 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.058819 4947 scope.go:122] "RemoveContainer" containerID="88bc1a67ee3173222e78dab6a51f0539babfda2e3f573e6e938ae9f27f8d2136" Nov 03 17:15:18.059324 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.059290 4947 scope.go:122] "RemoveContainer" containerID="def2b245503c7613091672b5c052d4e82d87b8f821152e81aa034295b8d63871" Nov 03 17:15:18.059815 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:18.059789 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj_odh-ai-gateway-infra(2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" podUID="2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd" Nov 03 17:15:18.061783 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.061763 4947 generic.go:356] "Generic (PLEG): container finished" podID="8afe859d-ebfc-4d31-9697-83198266d84a" containerID="a51b6d1d3da2ef40ce25d2c9ce74c6744c0002fd49b1428eb86eafc42c273cf1" exitCode=60 Nov 03 17:15:18.061870 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.061829 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" event={"ID":"8afe859d-ebfc-4d31-9697-83198266d84a","Type":"ContainerDied","Data":"a51b6d1d3da2ef40ce25d2c9ce74c6744c0002fd49b1428eb86eafc42c273cf1"} Nov 03 17:15:18.062271 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.062249 4947 scope.go:122] "RemoveContainer" containerID="a51b6d1d3da2ef40ce25d2c9ce74c6744c0002fd49b1428eb86eafc42c273cf1" Nov 03 17:15:18.062658 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:18.062637 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl_odh-ai-gateway-infra(8afe859d-ebfc-4d31-9697-83198266d84a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" podUID="8afe859d-ebfc-4d31-9697-83198266d84a" Nov 03 17:15:18.063825 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.063801 4947 generic.go:356] "Generic (PLEG): container finished" podID="6ac6d29c-ce79-4cd7-9dac-07836567e67f" containerID="641eef25f9e32e01c9f454f230ca12595ee55ff651408a126c78990ec6547ef3" exitCode=60 Nov 03 17:15:18.064086 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.063874 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" event={"ID":"6ac6d29c-ce79-4cd7-9dac-07836567e67f","Type":"ContainerDied","Data":"641eef25f9e32e01c9f454f230ca12595ee55ff651408a126c78990ec6547ef3"} Nov 03 17:15:18.064305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.064288 4947 scope.go:122] "RemoveContainer" containerID="641eef25f9e32e01c9f454f230ca12595ee55ff651408a126c78990ec6547ef3" Nov 03 17:15:18.064653 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:18.064633 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr_odh-ai-gateway-infra(6ac6d29c-ce79-4cd7-9dac-07836567e67f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" podUID="6ac6d29c-ce79-4cd7-9dac-07836567e67f" Nov 03 17:15:18.065455 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.065438 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" Nov 03 17:15:18.065542 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.065435 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9" event={"ID":"32f33b7a-8400-4518-b408-6dd563d84820","Type":"ContainerDied","Data":"99c302b33c81d99e761dd66ebbff671aaac01bbb668568e692e0acf33f754160"} Nov 03 17:15:18.076136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.076114 4947 scope.go:122] "RemoveContainer" containerID="906d88f01a6ca30ec07be91cf3a3cc4d89a8a9719c95e5d00dfe47b474c339af" Nov 03 17:15:18.090978 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.090954 4947 scope.go:122] "RemoveContainer" containerID="40ba2b1375314074c6fee4130582e60d163d42c384dbc764dc33d1cf097945c1" Nov 03 17:15:18.108167 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.108142 4947 scope.go:122] "RemoveContainer" containerID="91211d79d7dfd6a6e6df1adbd463436917838e9f1e52da093c1861f746112488" Nov 03 17:15:18.229908 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.229878 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9"] Nov 03 17:15:18.234785 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.234755 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895435-kkpf9"] Nov 03 17:15:18.236626 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.236606 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" Nov 03 17:15:18.395233 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.395196 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5122b48b-0b08-4047-9245-a9c23e6a7187-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5122b48b-0b08-4047-9245-a9c23e6a7187-maas-api-service-ca\") pod \"5122b48b-0b08-4047-9245-a9c23e6a7187\" (UID: \"5122b48b-0b08-4047-9245-a9c23e6a7187\") " Nov 03 17:15:18.395423 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.395325 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5122b48b-0b08-4047-9245-a9c23e6a7187-kube-api-access-djs4j\" (UniqueName: \"kubernetes.io/projected/5122b48b-0b08-4047-9245-a9c23e6a7187-kube-api-access-djs4j\") pod \"5122b48b-0b08-4047-9245-a9c23e6a7187\" (UID: \"5122b48b-0b08-4047-9245-a9c23e6a7187\") " Nov 03 17:15:18.395567 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.395544 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5122b48b-0b08-4047-9245-a9c23e6a7187-maas-api-service-ca" pod "5122b48b-0b08-4047-9245-a9c23e6a7187" (UID: "5122b48b-0b08-4047-9245-a9c23e6a7187"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:18.396124 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.396107 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5122b48b-0b08-4047-9245-a9c23e6a7187-maas-api-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:18.397928 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.397896 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5122b48b-0b08-4047-9245-a9c23e6a7187-kube-api-access-djs4j" pod "5122b48b-0b08-4047-9245-a9c23e6a7187" (UID: "5122b48b-0b08-4047-9245-a9c23e6a7187"). InnerVolumeSpecName "kube-api-access-djs4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:18.497856 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.497817 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djs4j\" (UniqueName: \"kubernetes.io/projected/5122b48b-0b08-4047-9245-a9c23e6a7187-kube-api-access-djs4j\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:18.558980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.558949 4947 scope.go:122] "RemoveContainer" containerID="a2062fb0c679ff393527a1fa61ebf1f359c3918866de934e46a2beeb374b92ab" Nov 03 17:15:18.669864 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:18.667040 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32f33b7a-8400-4518-b408-6dd563d84820" path="/var/lib/kubelet/pods/32f33b7a-8400-4518-b408-6dd563d84820/volumes" Nov 03 17:15:19.079330 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.079239 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" event={"ID":"705f3454-7a40-4665-9d0d-0beec2f25e31","Type":"ContainerStarted","Data":"ba96e52d5fa62ec6f22fb84d1c08cbd68662b9c4e73ddfbed3320c89120216ef"} Nov 03 17:15:19.081166 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.081139 4947 generic.go:356] "Generic (PLEG): container finished" podID="41406716-28a0-4b76-aeb5-c2a0a9b6292e" containerID="c86da877e3d3640cfba0f97bf09cc5203bba70b17449a65e983760b0345d1b28" exitCode=60 Nov 03 17:15:19.081341 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.081206 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" event={"ID":"41406716-28a0-4b76-aeb5-c2a0a9b6292e","Type":"ContainerDied","Data":"c86da877e3d3640cfba0f97bf09cc5203bba70b17449a65e983760b0345d1b28"} Nov 03 17:15:19.081341 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.081262 4947 scope.go:122] "RemoveContainer" containerID="693102659ddc0080d45e6c2caabae78e080faf46f657b2f7b66a77efb89369ae" Nov 03 17:15:19.081745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.081722 4947 scope.go:122] "RemoveContainer" containerID="c86da877e3d3640cfba0f97bf09cc5203bba70b17449a65e983760b0345d1b28" Nov 03 17:15:19.082651 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:19.082159 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7_odh-ai-gateway-infra(41406716-28a0-4b76-aeb5-c2a0a9b6292e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" podUID="41406716-28a0-4b76-aeb5-c2a0a9b6292e" Nov 03 17:15:19.083004 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.082972 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" event={"ID":"5122b48b-0b08-4047-9245-a9c23e6a7187","Type":"ContainerDied","Data":"59c1af1c9074df45bf20ff6dab45d3a849a13149761bebb743df6b0ad0449a0d"} Nov 03 17:15:19.083119 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.082978 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b" Nov 03 17:15:19.095225 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.095205 4947 scope.go:122] "RemoveContainer" containerID="5e0adfc79180711e4bcb3d318378720d9fc1637299ca4da88c44cad6762bfe79" Nov 03 17:15:19.157332 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.157288 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl"] Nov 03 17:15:19.203685 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.203638 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr"] Nov 03 17:15:19.212408 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.212339 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b"] Nov 03 17:15:19.217639 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.217606 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-a9426b-29895435-nqk8b"] Nov 03 17:15:19.286742 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.285907 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj"] Nov 03 17:15:19.380155 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.380123 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" Nov 03 17:15:19.479316 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.479286 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" Nov 03 17:15:19.516364 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.516334 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8afe859d-ebfc-4d31-9697-83198266d84a-kube-api-access-lxkhx\" (UniqueName: \"kubernetes.io/projected/8afe859d-ebfc-4d31-9697-83198266d84a-kube-api-access-lxkhx\") pod \"8afe859d-ebfc-4d31-9697-83198266d84a\" (UID: \"8afe859d-ebfc-4d31-9697-83198266d84a\") " Nov 03 17:15:19.516525 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.516412 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8afe859d-ebfc-4d31-9697-83198266d84a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afe859d-ebfc-4d31-9697-83198266d84a-maas-api-service-ca\") pod \"8afe859d-ebfc-4d31-9697-83198266d84a\" (UID: \"8afe859d-ebfc-4d31-9697-83198266d84a\") " Nov 03 17:15:19.516856 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.516828 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8afe859d-ebfc-4d31-9697-83198266d84a-maas-api-service-ca" pod "8afe859d-ebfc-4d31-9697-83198266d84a" (UID: "8afe859d-ebfc-4d31-9697-83198266d84a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:19.517149 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.517131 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8afe859d-ebfc-4d31-9697-83198266d84a-maas-api-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:19.518611 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.518588 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8afe859d-ebfc-4d31-9697-83198266d84a-kube-api-access-lxkhx" pod "8afe859d-ebfc-4d31-9697-83198266d84a" (UID: "8afe859d-ebfc-4d31-9697-83198266d84a"). InnerVolumeSpecName "kube-api-access-lxkhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:19.542600 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.542574 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" Nov 03 17:15:19.617841 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.617801 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd-kube-api-access-xkncs\" (UniqueName: \"kubernetes.io/projected/2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd-kube-api-access-xkncs\") pod \"2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd\" (UID: \"2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd\") " Nov 03 17:15:19.618041 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.617979 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac6d29c-ce79-4cd7-9dac-07836567e67f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac6d29c-ce79-4cd7-9dac-07836567e67f-maas-api-service-ca\") pod \"6ac6d29c-ce79-4cd7-9dac-07836567e67f\" (UID: \"6ac6d29c-ce79-4cd7-9dac-07836567e67f\") " Nov 03 17:15:19.618041 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.618026 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd-maas-api-service-ca\") pod \"2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd\" (UID: \"2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd\") " Nov 03 17:15:19.618290 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.618276 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac6d29c-ce79-4cd7-9dac-07836567e67f-kube-api-access-gtkfd\" (UniqueName: \"kubernetes.io/projected/6ac6d29c-ce79-4cd7-9dac-07836567e67f-kube-api-access-gtkfd\") pod \"6ac6d29c-ce79-4cd7-9dac-07836567e67f\" (UID: \"6ac6d29c-ce79-4cd7-9dac-07836567e67f\") " Nov 03 17:15:19.618794 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.618774 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxkhx\" (UniqueName: \"kubernetes.io/projected/8afe859d-ebfc-4d31-9697-83198266d84a-kube-api-access-lxkhx\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:19.619208 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.619180 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd-maas-api-service-ca" pod "2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd" (UID: "2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:19.619493 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.619457 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac6d29c-ce79-4cd7-9dac-07836567e67f-maas-api-service-ca" pod "6ac6d29c-ce79-4cd7-9dac-07836567e67f" (UID: "6ac6d29c-ce79-4cd7-9dac-07836567e67f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:19.621261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.621237 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac6d29c-ce79-4cd7-9dac-07836567e67f-kube-api-access-gtkfd" pod "6ac6d29c-ce79-4cd7-9dac-07836567e67f" (UID: "6ac6d29c-ce79-4cd7-9dac-07836567e67f"). InnerVolumeSpecName "kube-api-access-gtkfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:19.621418 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.621402 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd-kube-api-access-xkncs" pod "2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd" (UID: "2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd"). InnerVolumeSpecName "kube-api-access-xkncs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:19.720460 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.720432 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtkfd\" (UniqueName: \"kubernetes.io/projected/6ac6d29c-ce79-4cd7-9dac-07836567e67f-kube-api-access-gtkfd\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:19.720460 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.720459 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkncs\" (UniqueName: \"kubernetes.io/projected/2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd-kube-api-access-xkncs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:19.720460 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.720471 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac6d29c-ce79-4cd7-9dac-07836567e67f-maas-api-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:19.720751 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:19.720485 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd-maas-api-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:20.093430 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.093399 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" Nov 03 17:15:20.093892 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.093392 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr" event={"ID":"6ac6d29c-ce79-4cd7-9dac-07836567e67f","Type":"ContainerDied","Data":"a8ddbe4c15c7f4853c35c1cd12bf9a19f39853ec0cb142e2ea26702b17f7df30"} Nov 03 17:15:20.093892 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.093582 4947 scope.go:122] "RemoveContainer" containerID="641eef25f9e32e01c9f454f230ca12595ee55ff651408a126c78990ec6547ef3" Nov 03 17:15:20.095483 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.095457 4947 generic.go:356] "Generic (PLEG): container finished" podID="705f3454-7a40-4665-9d0d-0beec2f25e31" containerID="ba96e52d5fa62ec6f22fb84d1c08cbd68662b9c4e73ddfbed3320c89120216ef" exitCode=60 Nov 03 17:15:20.095648 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.095528 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" event={"ID":"705f3454-7a40-4665-9d0d-0beec2f25e31","Type":"ContainerDied","Data":"ba96e52d5fa62ec6f22fb84d1c08cbd68662b9c4e73ddfbed3320c89120216ef"} Nov 03 17:15:20.096054 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.096022 4947 scope.go:122] "RemoveContainer" containerID="ba96e52d5fa62ec6f22fb84d1c08cbd68662b9c4e73ddfbed3320c89120216ef" Nov 03 17:15:20.096380 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:20.096360 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk_odh-ai-gateway-infra(705f3454-7a40-4665-9d0d-0beec2f25e31)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" podUID="705f3454-7a40-4665-9d0d-0beec2f25e31" Nov 03 17:15:20.100232 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.100205 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" Nov 03 17:15:20.100359 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.100231 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj" event={"ID":"2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd","Type":"ContainerDied","Data":"efc572e653260146c87f86b71ada51b264c45163e96e09537660b9b2f6bbe1a1"} Nov 03 17:15:20.101813 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.101774 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" event={"ID":"8afe859d-ebfc-4d31-9697-83198266d84a","Type":"ContainerDied","Data":"f84ff9c91a83fe1baac55c6d819819f2dc648be0995df8cb38c48213dbac0022"} Nov 03 17:15:20.101937 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.101868 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl" Nov 03 17:15:20.107966 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.107945 4947 scope.go:122] "RemoveContainer" containerID="a2062fb0c679ff393527a1fa61ebf1f359c3918866de934e46a2beeb374b92ab" Nov 03 17:15:20.125076 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.125054 4947 scope.go:122] "RemoveContainer" containerID="def2b245503c7613091672b5c052d4e82d87b8f821152e81aa034295b8d63871" Nov 03 17:15:20.138138 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.138112 4947 scope.go:122] "RemoveContainer" containerID="a51b6d1d3da2ef40ce25d2c9ce74c6744c0002fd49b1428eb86eafc42c273cf1" Nov 03 17:15:20.139652 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.139627 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk"] Nov 03 17:15:20.183559 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.183527 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7"] Nov 03 17:15:20.223053 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.223018 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr"] Nov 03 17:15:20.229791 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.229765 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-7519b7-29895435-p8vpr"] Nov 03 17:15:20.384167 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.384135 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj"] Nov 03 17:15:20.388768 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.388717 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3b6f35-29895435-pjhtj"] Nov 03 17:15:20.406908 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.406879 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl"] Nov 03 17:15:20.412380 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.412351 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3a2fa2-29895435-m22pl"] Nov 03 17:15:20.459175 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.459151 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" Nov 03 17:15:20.554235 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.554200 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41406716-28a0-4b76-aeb5-c2a0a9b6292e-kube-api-access-kfbdc\" (UniqueName: \"kubernetes.io/projected/41406716-28a0-4b76-aeb5-c2a0a9b6292e-kube-api-access-kfbdc\") pod \"41406716-28a0-4b76-aeb5-c2a0a9b6292e\" (UID: \"41406716-28a0-4b76-aeb5-c2a0a9b6292e\") " Nov 03 17:15:20.554457 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.554292 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41406716-28a0-4b76-aeb5-c2a0a9b6292e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41406716-28a0-4b76-aeb5-c2a0a9b6292e-maas-api-service-ca\") pod \"41406716-28a0-4b76-aeb5-c2a0a9b6292e\" (UID: \"41406716-28a0-4b76-aeb5-c2a0a9b6292e\") " Nov 03 17:15:20.554760 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.554734 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41406716-28a0-4b76-aeb5-c2a0a9b6292e-maas-api-service-ca" pod "41406716-28a0-4b76-aeb5-c2a0a9b6292e" (UID: "41406716-28a0-4b76-aeb5-c2a0a9b6292e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:20.555083 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.555065 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41406716-28a0-4b76-aeb5-c2a0a9b6292e-maas-api-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:20.556531 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.556510 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41406716-28a0-4b76-aeb5-c2a0a9b6292e-kube-api-access-kfbdc" pod "41406716-28a0-4b76-aeb5-c2a0a9b6292e" (UID: "41406716-28a0-4b76-aeb5-c2a0a9b6292e"). InnerVolumeSpecName "kube-api-access-kfbdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:20.575947 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.575915 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd" path="/var/lib/kubelet/pods/2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd/volumes" Nov 03 17:15:20.576643 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.576623 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5122b48b-0b08-4047-9245-a9c23e6a7187" path="/var/lib/kubelet/pods/5122b48b-0b08-4047-9245-a9c23e6a7187/volumes" Nov 03 17:15:20.577253 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.577234 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac6d29c-ce79-4cd7-9dac-07836567e67f" path="/var/lib/kubelet/pods/6ac6d29c-ce79-4cd7-9dac-07836567e67f/volumes" Nov 03 17:15:20.578053 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.578032 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8afe859d-ebfc-4d31-9697-83198266d84a" path="/var/lib/kubelet/pods/8afe859d-ebfc-4d31-9697-83198266d84a/volumes" Nov 03 17:15:20.657690 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:20.657605 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfbdc\" (UniqueName: \"kubernetes.io/projected/41406716-28a0-4b76-aeb5-c2a0a9b6292e-kube-api-access-kfbdc\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:21.115322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:21.115284 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" event={"ID":"41406716-28a0-4b76-aeb5-c2a0a9b6292e","Type":"ContainerDied","Data":"2addcc855f328a48a4ab75ce6199f04ae0da13d5614ae470f1bd85ae873383ee"} Nov 03 17:15:21.115322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:21.115315 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7" Nov 03 17:15:21.115789 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:21.115350 4947 scope.go:122] "RemoveContainer" containerID="c86da877e3d3640cfba0f97bf09cc5203bba70b17449a65e983760b0345d1b28" Nov 03 17:15:21.205551 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:21.205516 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7"] Nov 03 17:15:21.214623 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:21.214588 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-9a80f0-29895435-bxmb7"] Nov 03 17:15:21.287510 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:21.287485 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" Nov 03 17:15:21.470228 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:21.470130 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/705f3454-7a40-4665-9d0d-0beec2f25e31-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/705f3454-7a40-4665-9d0d-0beec2f25e31-maas-api-service-ca\") pod \"705f3454-7a40-4665-9d0d-0beec2f25e31\" (UID: \"705f3454-7a40-4665-9d0d-0beec2f25e31\") " Nov 03 17:15:21.470424 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:21.470381 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705f3454-7a40-4665-9d0d-0beec2f25e31-kube-api-access-hz4dr\" (UniqueName: \"kubernetes.io/projected/705f3454-7a40-4665-9d0d-0beec2f25e31-kube-api-access-hz4dr\") pod \"705f3454-7a40-4665-9d0d-0beec2f25e31\" (UID: \"705f3454-7a40-4665-9d0d-0beec2f25e31\") " Nov 03 17:15:21.470564 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:21.470540 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705f3454-7a40-4665-9d0d-0beec2f25e31-maas-api-service-ca" pod "705f3454-7a40-4665-9d0d-0beec2f25e31" (UID: "705f3454-7a40-4665-9d0d-0beec2f25e31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:15:21.471088 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:21.471069 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/705f3454-7a40-4665-9d0d-0beec2f25e31-maas-api-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:21.472908 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:21.472883 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705f3454-7a40-4665-9d0d-0beec2f25e31-kube-api-access-hz4dr" pod "705f3454-7a40-4665-9d0d-0beec2f25e31" (UID: "705f3454-7a40-4665-9d0d-0beec2f25e31"). InnerVolumeSpecName "kube-api-access-hz4dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:15:21.572507 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:21.572461 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz4dr\" (UniqueName: \"kubernetes.io/projected/705f3454-7a40-4665-9d0d-0beec2f25e31-kube-api-access-hz4dr\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:15:21.938267 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:21.938221 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:15:22.139230 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:22.139189 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" Nov 03 17:15:22.139733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:22.139309 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk" event={"ID":"705f3454-7a40-4665-9d0d-0beec2f25e31","Type":"ContainerDied","Data":"39ec9b2cbd8deb6a6a99f01217e6226d3a10975d36f56cb53b95b8c07e5e4043"} Nov 03 17:15:22.139733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:22.139428 4947 scope.go:122] "RemoveContainer" containerID="ba96e52d5fa62ec6f22fb84d1c08cbd68662b9c4e73ddfbed3320c89120216ef" Nov 03 17:15:22.199249 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:22.199167 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk"] Nov 03 17:15:22.203435 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:22.203406 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895435-sdtkk"] Nov 03 17:15:22.567597 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:22.567565 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41406716-28a0-4b76-aeb5-c2a0a9b6292e" path="/var/lib/kubelet/pods/41406716-28a0-4b76-aeb5-c2a0a9b6292e/volumes" Nov 03 17:15:22.568148 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:22.568132 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="705f3454-7a40-4665-9d0d-0beec2f25e31" path="/var/lib/kubelet/pods/705f3454-7a40-4665-9d0d-0beec2f25e31/volumes" Nov 03 17:15:23.807759 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.806757 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb"] Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808026 4947 cpu_manager.go:414] "containerMap: removing container" podUID="5122b48b-0b08-4047-9245-a9c23e6a7187" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808049 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5122b48b-0b08-4047-9245-a9c23e6a7187" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808065 4947 cpu_manager.go:414] "containerMap: removing container" podUID="6ac6d29c-ce79-4cd7-9dac-07836567e67f" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808073 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac6d29c-ce79-4cd7-9dac-07836567e67f" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808087 4947 cpu_manager.go:414] "containerMap: removing container" podUID="41406716-28a0-4b76-aeb5-c2a0a9b6292e" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808096 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41406716-28a0-4b76-aeb5-c2a0a9b6292e" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808111 4947 cpu_manager.go:414] "containerMap: removing container" podUID="2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808118 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808134 4947 cpu_manager.go:414] "containerMap: removing container" podUID="2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808140 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808157 4947 cpu_manager.go:414] "containerMap: removing container" podUID="41406716-28a0-4b76-aeb5-c2a0a9b6292e" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808164 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41406716-28a0-4b76-aeb5-c2a0a9b6292e" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808174 4947 cpu_manager.go:414] "containerMap: removing container" podUID="32f33b7a-8400-4518-b408-6dd563d84820" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808181 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f33b7a-8400-4518-b408-6dd563d84820" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808191 4947 cpu_manager.go:414] "containerMap: removing container" podUID="32f33b7a-8400-4518-b408-6dd563d84820" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808197 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f33b7a-8400-4518-b408-6dd563d84820" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808207 4947 cpu_manager.go:414] "containerMap: removing container" podUID="6ac6d29c-ce79-4cd7-9dac-07836567e67f" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808214 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac6d29c-ce79-4cd7-9dac-07836567e67f" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808222 4947 cpu_manager.go:414] "containerMap: removing container" podUID="5122b48b-0b08-4047-9245-a9c23e6a7187" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808229 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5122b48b-0b08-4047-9245-a9c23e6a7187" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808240 4947 cpu_manager.go:414] "containerMap: removing container" podUID="5122b48b-0b08-4047-9245-a9c23e6a7187" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808246 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5122b48b-0b08-4047-9245-a9c23e6a7187" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808260 4947 cpu_manager.go:414] "containerMap: removing container" podUID="705f3454-7a40-4665-9d0d-0beec2f25e31" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808267 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705f3454-7a40-4665-9d0d-0beec2f25e31" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808285 4947 cpu_manager.go:414] "containerMap: removing container" podUID="8afe859d-ebfc-4d31-9697-83198266d84a" containerName="cleanup" Nov 03 17:15:23.808281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808292 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8afe859d-ebfc-4d31-9697-83198266d84a" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808302 4947 cpu_manager.go:414] "containerMap: removing container" podUID="8afe859d-ebfc-4d31-9697-83198266d84a" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808310 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8afe859d-ebfc-4d31-9697-83198266d84a" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808346 4947 cpu_manager.go:414] "containerMap: removing container" podUID="705f3454-7a40-4665-9d0d-0beec2f25e31" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808353 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705f3454-7a40-4665-9d0d-0beec2f25e31" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808378 4947 cpu_manager.go:414] "containerMap: removing container" podUID="705f3454-7a40-4665-9d0d-0beec2f25e31" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808385 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705f3454-7a40-4665-9d0d-0beec2f25e31" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808575 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808590 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="41406716-28a0-4b76-aeb5-c2a0a9b6292e" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808597 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="705f3454-7a40-4665-9d0d-0beec2f25e31" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808608 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="8afe859d-ebfc-4d31-9697-83198266d84a" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808616 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac6d29c-ce79-4cd7-9dac-07836567e67f" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808624 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="705f3454-7a40-4665-9d0d-0beec2f25e31" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808635 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f33b7a-8400-4518-b408-6dd563d84820" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808645 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="5122b48b-0b08-4047-9245-a9c23e6a7187" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808653 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="41406716-28a0-4b76-aeb5-c2a0a9b6292e" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808664 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808674 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="41406716-28a0-4b76-aeb5-c2a0a9b6292e" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808684 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f33b7a-8400-4518-b408-6dd563d84820" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808714 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f33b7a-8400-4518-b408-6dd563d84820" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808722 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac6d29c-ce79-4cd7-9dac-07836567e67f" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808731 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="5122b48b-0b08-4047-9245-a9c23e6a7187" containerName="cleanup" Nov 03 17:15:23.809721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.808743 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="8afe859d-ebfc-4d31-9697-83198266d84a" containerName="cleanup" Nov 03 17:15:23.854726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.838996 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:15:23.872723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.862628 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-24cb6d-serving-cert\"" Nov 03 17:15:23.873817 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.873787 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb"] Nov 03 17:15:23.894232 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.888122 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-openshift-service-ca\") pod \"maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb\" (UID: \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:15:23.894475 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.894455 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-maas-api-tls\") pod \"maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb\" (UID: \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:15:23.920735 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.894623 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-metrics-certs\") pod \"maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb\" (UID: \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:15:23.921176 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:23.921139 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxqcb\" (UniqueName: \"kubernetes.io/projected/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-kube-api-access-hxqcb\") pod \"maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb\" (UID: \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:15:24.084786 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:24.062332 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-maas-api-tls\") pod \"maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb\" (UID: \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:15:24.084786 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:24.062436 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-metrics-certs\") pod \"maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb\" (UID: \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:15:24.084786 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:24.062585 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqcb\" (UniqueName: \"kubernetes.io/projected/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-kube-api-access-hxqcb\") pod \"maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb\" (UID: \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:15:24.084786 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:24.063011 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-openshift-service-ca\") pod \"maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb\" (UID: \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:15:24.084786 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:24.063986 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-openshift-service-ca\") pod \"maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb\" (UID: \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:15:24.084786 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:24.064117 4947 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-serving-cert: secret "maas-api-e2e-inf-a-24cb6d-serving-cert" not found Nov 03 17:15:24.084786 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:24.064178 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-maas-api-tls podName:0fc48229-09ca-4ae5-9cb2-de8e8ea21cad nodeName:}" failed. No retries permitted until 2026-11-03 17:15:24.56415746 +0000 UTC m=+3668.153157408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-maas-api-tls") pod "maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" (UID: "0fc48229-09ca-4ae5-9cb2-de8e8ea21cad") : secret "maas-api-e2e-inf-a-24cb6d-serving-cert" not found Nov 03 17:15:24.136674 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:24.136633 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-metrics-certs\") pod \"maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb\" (UID: \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:15:24.179784 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:24.179750 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxqcb\" (UniqueName: \"kubernetes.io/projected/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-kube-api-access-hxqcb\") pod \"maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb\" (UID: \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:15:24.596150 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:24.596115 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-maas-api-tls\") pod \"maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb\" (UID: \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:15:24.613175 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:24.605388 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-maas-api-tls\") pod \"maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb\" (UID: \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:15:24.882772 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:24.878018 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:15:25.876631 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:25.876584 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb"] Nov 03 17:15:25.879614 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:15:25.879562 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc48229_09ca_4ae5_9cb2_de8e8ea21cad.slice/crio-ee3aacf78c2bdb4e2c05f67eab089a8e10bc41557772463465875bb459d5f4e1 WatchSource:0}: Error finding container ee3aacf78c2bdb4e2c05f67eab089a8e10bc41557772463465875bb459d5f4e1: Status 404 returned error can't find the container with id ee3aacf78c2bdb4e2c05f67eab089a8e10bc41557772463465875bb459d5f4e1 Nov 03 17:15:25.881945 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:25.881924 4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:15:26.489220 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:26.489160 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" event={"ID":"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad","Type":"ContainerStarted","Data":"2a61e72958c4d631f5ac37bc753c2fe6acfab3bfce2068a1893f18c880afb9bb"} Nov 03 17:15:26.489760 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:26.489231 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" event={"ID":"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad","Type":"ContainerStarted","Data":"ee3aacf78c2bdb4e2c05f67eab089a8e10bc41557772463465875bb459d5f4e1"} Nov 03 17:15:26.489760 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:26.489257 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:15:26.539061 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:26.539002 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" podStartSLOduration=3.060796875 podStartE2EDuration="3.538987384s" totalImagesPullingTime="478.190509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:26.535498804 +0000 UTC m=+3670.124498772" watchObservedRunningTime="2026-11-03 17:15:26.538987384 +0000 UTC m=+3670.127987352" Nov 03 17:15:30.131122 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:30.131086 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:15:32.503632 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.503597 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:15:32.616670 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.616594 4947 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod64232765-b0cc-498c-9041-048ed8285ad9"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod64232765-b0cc-498c-9041-048ed8285ad9] : Timed out while waiting for systemd to remove kubepods-burstable-pod64232765_b0cc_498c_9041_048ed8285ad9.slice" Nov 03 17:15:32.616670 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.616646 4947 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="64232765-b0cc-498c-9041-048ed8285ad9" err="failed to delete cgroup paths for [kubepods burstable pod64232765-b0cc-498c-9041-048ed8285ad9] : unable to destroy cgroup paths for cgroup [kubepods burstable pod64232765-b0cc-498c-9041-048ed8285ad9] : Timed out while waiting for systemd to remove kubepods-burstable-pod64232765_b0cc_498c_9041_048ed8285ad9.slice" Nov 03 17:15:32.832186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.832107 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4"] Nov 03 17:15:32.832977 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.832960 4947 cpu_manager.go:414] "containerMap: removing container" podUID="6ac6d29c-ce79-4cd7-9dac-07836567e67f" containerName="cleanup" Nov 03 17:15:32.833034 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.832978 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac6d29c-ce79-4cd7-9dac-07836567e67f" containerName="cleanup" Nov 03 17:15:32.833034 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.832994 4947 cpu_manager.go:414] "containerMap: removing container" podUID="2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd" containerName="cleanup" Nov 03 17:15:32.833034 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.832998 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd" containerName="cleanup" Nov 03 17:15:32.833034 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.833009 4947 cpu_manager.go:414] "containerMap: removing container" podUID="41406716-28a0-4b76-aeb5-c2a0a9b6292e" containerName="cleanup" Nov 03 17:15:32.833034 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.833013 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41406716-28a0-4b76-aeb5-c2a0a9b6292e" containerName="cleanup" Nov 03 17:15:32.833034 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.833020 4947 cpu_manager.go:414] "containerMap: removing container" podUID="32f33b7a-8400-4518-b408-6dd563d84820" containerName="cleanup" Nov 03 17:15:32.833034 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.833026 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f33b7a-8400-4518-b408-6dd563d84820" containerName="cleanup" Nov 03 17:15:32.833243 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.833039 4947 cpu_manager.go:414] "containerMap: removing container" podUID="8afe859d-ebfc-4d31-9697-83198266d84a" containerName="cleanup" Nov 03 17:15:32.833243 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.833043 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8afe859d-ebfc-4d31-9697-83198266d84a" containerName="cleanup" Nov 03 17:15:32.833243 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.833150 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac6d29c-ce79-4cd7-9dac-07836567e67f" containerName="cleanup" Nov 03 17:15:32.833243 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.833156 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="5122b48b-0b08-4047-9245-a9c23e6a7187" containerName="cleanup" Nov 03 17:15:32.833243 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.833167 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="8afe859d-ebfc-4d31-9697-83198266d84a" containerName="cleanup" Nov 03 17:15:32.833243 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.833174 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8a7026-f9bd-4ded-abcb-1e3c8ab733fd" containerName="cleanup" Nov 03 17:15:32.833243 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.833183 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="705f3454-7a40-4665-9d0d-0beec2f25e31" containerName="cleanup" Nov 03 17:15:32.838345 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.838327 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:15:32.855281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.855249 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-44736d-serving-cert\"" Nov 03 17:15:32.861073 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.861045 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4"] Nov 03 17:15:32.928999 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.928966 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ddeb2f-f94f-4012-a987-259a185e19aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4\" (UID: \"23ddeb2f-f94f-4012-a987-259a185e19aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:15:32.929190 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.929173 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfvnc\" (UniqueName: \"kubernetes.io/projected/23ddeb2f-f94f-4012-a987-259a185e19aa-kube-api-access-rfvnc\") pod \"maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4\" (UID: \"23ddeb2f-f94f-4012-a987-259a185e19aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:15:32.929415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.929388 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ddeb2f-f94f-4012-a987-259a185e19aa-metrics-certs\") pod \"maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4\" (UID: \"23ddeb2f-f94f-4012-a987-259a185e19aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:15:32.929561 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:32.929535 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ddeb2f-f94f-4012-a987-259a185e19aa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4\" (UID: \"23ddeb2f-f94f-4012-a987-259a185e19aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:15:33.033740 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:33.032528 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ddeb2f-f94f-4012-a987-259a185e19aa-metrics-certs\") pod \"maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4\" (UID: \"23ddeb2f-f94f-4012-a987-259a185e19aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:15:33.033740 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:33.032667 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ddeb2f-f94f-4012-a987-259a185e19aa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4\" (UID: \"23ddeb2f-f94f-4012-a987-259a185e19aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:15:33.033740 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:33.032733 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ddeb2f-f94f-4012-a987-259a185e19aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4\" (UID: \"23ddeb2f-f94f-4012-a987-259a185e19aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:15:33.038281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:33.038184 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ddeb2f-f94f-4012-a987-259a185e19aa-openshift-service-ca\") pod \"maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4\" (UID: \"23ddeb2f-f94f-4012-a987-259a185e19aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:15:33.043937 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:33.043875 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ddeb2f-f94f-4012-a987-259a185e19aa-metrics-certs\") pod \"maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4\" (UID: \"23ddeb2f-f94f-4012-a987-259a185e19aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:15:33.052645 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:33.046866 4947 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-serving-cert: secret "maas-api-e2e-inf-a-44736d-serving-cert" not found Nov 03 17:15:33.052645 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:15:33.046952 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23ddeb2f-f94f-4012-a987-259a185e19aa-maas-api-tls podName:23ddeb2f-f94f-4012-a987-259a185e19aa nodeName:}" failed. No retries permitted until 2026-11-03 17:15:33.546924682 +0000 UTC m=+3677.135924645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/23ddeb2f-f94f-4012-a987-259a185e19aa-maas-api-tls") pod "maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" (UID: "23ddeb2f-f94f-4012-a987-259a185e19aa") : secret "maas-api-e2e-inf-a-44736d-serving-cert" not found Nov 03 17:15:33.052645 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:33.047430 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfvnc\" (UniqueName: \"kubernetes.io/projected/23ddeb2f-f94f-4012-a987-259a185e19aa-kube-api-access-rfvnc\") pod \"maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4\" (UID: \"23ddeb2f-f94f-4012-a987-259a185e19aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:15:33.090380 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:33.090262 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfvnc\" (UniqueName: \"kubernetes.io/projected/23ddeb2f-f94f-4012-a987-259a185e19aa-kube-api-access-rfvnc\") pod \"maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4\" (UID: \"23ddeb2f-f94f-4012-a987-259a185e19aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:15:33.555911 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:33.555872 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ddeb2f-f94f-4012-a987-259a185e19aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4\" (UID: \"23ddeb2f-f94f-4012-a987-259a185e19aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:15:33.558561 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:33.558538 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ddeb2f-f94f-4012-a987-259a185e19aa-maas-api-tls\") pod \"maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4\" (UID: \"23ddeb2f-f94f-4012-a987-259a185e19aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:15:33.754448 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:33.754412 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:15:34.120325 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:34.120292 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4"] Nov 03 17:15:34.120504 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:15:34.120482 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ddeb2f_f94f_4012_a987_259a185e19aa.slice/crio-44084e7ebd711d3e6e19c2a0e90b550ff88200698e8f037f099f7e93239fc4d8 WatchSource:0}: Error finding container 44084e7ebd711d3e6e19c2a0e90b550ff88200698e8f037f099f7e93239fc4d8: Status 404 returned error can't find the container with id 44084e7ebd711d3e6e19c2a0e90b550ff88200698e8f037f099f7e93239fc4d8 Nov 03 17:15:34.555245 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:34.555217 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" event={"ID":"23ddeb2f-f94f-4012-a987-259a185e19aa","Type":"ContainerStarted","Data":"44084e7ebd711d3e6e19c2a0e90b550ff88200698e8f037f099f7e93239fc4d8"} Nov 03 17:15:35.564674 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:35.564634 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" event={"ID":"23ddeb2f-f94f-4012-a987-259a185e19aa","Type":"ContainerStarted","Data":"fcd897d7d58db82d16cec1dc1eabfa815893cf225bd9aad16c05a46e2509ece8"} Nov 03 17:15:35.565156 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:35.564736 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:15:35.614221 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:35.614158 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" podStartSLOduration=3.263073264 podStartE2EDuration="3.614141448s" totalImagesPullingTime="351.068184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:35.610500156 +0000 UTC m=+3679.199500123" watchObservedRunningTime="2026-11-03 17:15:35.614141448 +0000 UTC m=+3679.203141442" Nov 03 17:15:41.589031 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:41.588998 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:15:42.731488 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:42.731447 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t"] Nov 03 17:15:42.737849 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:42.737813 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" Nov 03 17:15:42.761134 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:42.761089 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t"] Nov 03 17:15:42.761591 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:42.761570 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-fc40cc-serving-cert\"" Nov 03 17:15:42.926720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:42.922425 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/769d68a5-7504-4c6a-9db1-355bff4c7ef8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t\" (UID: \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" Nov 03 17:15:42.926720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:42.922505 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv5wr\" (UniqueName: \"kubernetes.io/projected/769d68a5-7504-4c6a-9db1-355bff4c7ef8-kube-api-access-hv5wr\") pod \"maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t\" (UID: \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" Nov 03 17:15:42.926720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:42.922612 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/769d68a5-7504-4c6a-9db1-355bff4c7ef8-metrics-certs\") pod \"maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t\" (UID: \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" Nov 03 17:15:42.926720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:42.922717 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/769d68a5-7504-4c6a-9db1-355bff4c7ef8-maas-api-tls\") pod \"maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t\" (UID: \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" Nov 03 17:15:43.024077 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:43.024035 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/769d68a5-7504-4c6a-9db1-355bff4c7ef8-metrics-certs\") pod \"maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t\" (UID: \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" Nov 03 17:15:43.024274 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:43.024116 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/769d68a5-7504-4c6a-9db1-355bff4c7ef8-maas-api-tls\") pod \"maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t\" (UID: \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" Nov 03 17:15:43.024347 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:43.024315 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/769d68a5-7504-4c6a-9db1-355bff4c7ef8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t\" (UID: \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" Nov 03 17:15:43.024391 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:43.024357 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv5wr\" (UniqueName: \"kubernetes.io/projected/769d68a5-7504-4c6a-9db1-355bff4c7ef8-kube-api-access-hv5wr\") pod \"maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t\" (UID: \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" Nov 03 17:15:43.025938 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:43.025828 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/769d68a5-7504-4c6a-9db1-355bff4c7ef8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t\" (UID: \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" Nov 03 17:15:43.029125 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:43.029093 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/769d68a5-7504-4c6a-9db1-355bff4c7ef8-metrics-certs\") pod \"maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t\" (UID: \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" Nov 03 17:15:43.029260 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:43.029236 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/769d68a5-7504-4c6a-9db1-355bff4c7ef8-maas-api-tls\") pod \"maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t\" (UID: \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" Nov 03 17:15:43.049380 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:43.049353 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv5wr\" (UniqueName: \"kubernetes.io/projected/769d68a5-7504-4c6a-9db1-355bff4c7ef8-kube-api-access-hv5wr\") pod \"maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t\" (UID: \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" Nov 03 17:15:43.055281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:43.055259 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" Nov 03 17:15:43.373394 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:15:43.373354 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769d68a5_7504_4c6a_9db1_355bff4c7ef8.slice/crio-548818cc2d44d7c699aeedf49ee93d32bce463beb7a17473328cf05c7ae3f08a WatchSource:0}: Error finding container 548818cc2d44d7c699aeedf49ee93d32bce463beb7a17473328cf05c7ae3f08a: Status 404 returned error can't find the container with id 548818cc2d44d7c699aeedf49ee93d32bce463beb7a17473328cf05c7ae3f08a Nov 03 17:15:43.377306 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:43.377269 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t"] Nov 03 17:15:43.648012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:43.647915 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" event={"ID":"769d68a5-7504-4c6a-9db1-355bff4c7ef8","Type":"ContainerStarted","Data":"548818cc2d44d7c699aeedf49ee93d32bce463beb7a17473328cf05c7ae3f08a"} Nov 03 17:15:44.660045 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:44.660006 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" event={"ID":"769d68a5-7504-4c6a-9db1-355bff4c7ef8","Type":"ContainerStarted","Data":"b6894f3f36d8cd3264ad55436210533c36e3a035e68bc0b0f136d04a11e3f418"} Nov 03 17:15:44.660571 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:44.660106 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" Nov 03 17:15:44.709890 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:44.709836 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" podStartSLOduration=2.400764054 podStartE2EDuration="2.709819681s" totalImagesPullingTime="309.055627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:15:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:15:44.706817035 +0000 UTC m=+3688.295817015" watchObservedRunningTime="2026-11-03 17:15:44.709819681 +0000 UTC m=+3688.298819649" Nov 03 17:15:50.674646 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:50.674613 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" Nov 03 17:15:58.429517 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:58.429479 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw"] Nov 03 17:15:58.430451 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:58.430414 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" podUID="c844d192-95a9-4170-8af6-1b421fb533b5" containerName="istio-proxy" containerID="cri-o://b7096418373c3bdc542d7806ab775777fc04f1e1597b6e98aa7a110bd1ae14c0" gracePeriod=30 Nov 03 17:15:59.785802 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:59.785769 4947 patch_prober.go:29] interesting pod/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": read tcp 10.132.0.2:59802->10.132.0.241:15021: read: connection reset by peer" start-of-body= Nov 03 17:15:59.786267 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:59.785852 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" podUID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": read tcp 10.132.0.2:59802->10.132.0.241:15021: read: connection reset by peer" Nov 03 17:15:59.906177 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:59.906140 4947 generic.go:356] "Generic (PLEG): container finished" podID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" containerID="fda76cd77932b7436280edfeff09a7edebabc200d13dfb1bbcc274088f8be176" exitCode=0 Nov 03 17:15:59.906349 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:59.906210 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" event={"ID":"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d","Type":"ContainerDied","Data":"fda76cd77932b7436280edfeff09a7edebabc200d13dfb1bbcc274088f8be176"} Nov 03 17:15:59.906735 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:59.906711 4947 scope.go:122] "RemoveContainer" containerID="fda76cd77932b7436280edfeff09a7edebabc200d13dfb1bbcc274088f8be176" Nov 03 17:15:59.909001 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:59.908970 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:15:59.909110 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:59.909040 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:15:59.909110 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:15:59.909074 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:16:00.171622 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:00.167424 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:16:00.918054 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:00.917956 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" event={"ID":"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d","Type":"ContainerStarted","Data":"33028b9510697cbe5c77d92f8f584ffcab23ec66c60730bf4fc12f7c3c6a864a"} Nov 03 17:16:00.919916 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:00.919889 4947 generic.go:356] "Generic (PLEG): container finished" podID="a852b387-ca23-4e7a-b793-788feb393176" containerID="137cafd0e2519e9b29e6b9d0894445a8fd77e6b3aa47de07214a3dd16c740ae5" exitCode=0 Nov 03 17:16:00.920063 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:00.919924 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" event={"ID":"a852b387-ca23-4e7a-b793-788feb393176","Type":"ContainerDied","Data":"137cafd0e2519e9b29e6b9d0894445a8fd77e6b3aa47de07214a3dd16c740ae5"} Nov 03 17:16:00.920355 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:00.920337 4947 scope.go:122] "RemoveContainer" containerID="137cafd0e2519e9b29e6b9d0894445a8fd77e6b3aa47de07214a3dd16c740ae5" Nov 03 17:16:00.922714 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:00.922669 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:16:00.922834 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:00.922771 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:16:00.922834 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:00.922819 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:16:01.454773 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:01.450567 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:16:01.454773 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:01.450887 4947 patch_prober.go:29] interesting pod/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 03 17:16:01.454773 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:01.451040 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" podUID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 03 17:16:01.985725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:01.985437 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" event={"ID":"a852b387-ca23-4e7a-b793-788feb393176","Type":"ContainerStarted","Data":"6a9acb0c8e315371c88e962a89df8c015e929ad4cc049118a6fa34459283f8b6"} Nov 03 17:16:02.253101 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:02.253064 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:16:02.254161 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:02.254136 4947 patch_prober.go:29] interesting pod/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 17:16:02.254221 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:02.254198 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" podUID="a852b387-ca23-4e7a-b793-788feb393176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 17:16:02.451271 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:02.451220 4947 patch_prober.go:29] interesting pod/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 03 17:16:02.451619 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:02.451580 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" podUID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 03 17:16:03.253276 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:03.253234 4947 patch_prober.go:29] interesting pod/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 17:16:03.253876 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:03.253289 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" podUID="a852b387-ca23-4e7a-b793-788feb393176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 17:16:03.451748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:03.451396 4947 patch_prober.go:29] interesting pod/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 03 17:16:03.451748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:03.451460 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" podUID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 03 17:16:04.061581 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.061539 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:16:04.063741 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.063680 4947 generic.go:356] "Generic (PLEG): container finished" podID="c844d192-95a9-4170-8af6-1b421fb533b5" containerID="b7096418373c3bdc542d7806ab775777fc04f1e1597b6e98aa7a110bd1ae14c0" exitCode=0 Nov 03 17:16:04.063741 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.063728 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" event={"ID":"c844d192-95a9-4170-8af6-1b421fb533b5","Type":"ContainerDied","Data":"b7096418373c3bdc542d7806ab775777fc04f1e1597b6e98aa7a110bd1ae14c0"} Nov 03 17:16:04.063920 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.063789 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" event={"ID":"c844d192-95a9-4170-8af6-1b421fb533b5","Type":"ContainerDied","Data":"320c7470b4c2d8a598c8b779b1d22e28c8fc5758a9a5f0f20eb2f48bfc656ba4"} Nov 03 17:16:04.063920 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.063736 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw" Nov 03 17:16:04.063920 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.063817 4947 scope.go:122] "RemoveContainer" containerID="b7096418373c3bdc542d7806ab775777fc04f1e1597b6e98aa7a110bd1ae14c0" Nov 03 17:16:04.076086 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.076048 4947 scope.go:122] "RemoveContainer" containerID="b7096418373c3bdc542d7806ab775777fc04f1e1597b6e98aa7a110bd1ae14c0" Nov 03 17:16:04.076341 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:04.076321 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7096418373c3bdc542d7806ab775777fc04f1e1597b6e98aa7a110bd1ae14c0\": container with ID starting with b7096418373c3bdc542d7806ab775777fc04f1e1597b6e98aa7a110bd1ae14c0 not found: ID does not exist" containerID="b7096418373c3bdc542d7806ab775777fc04f1e1597b6e98aa7a110bd1ae14c0" Nov 03 17:16:04.076396 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.076352 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7096418373c3bdc542d7806ab775777fc04f1e1597b6e98aa7a110bd1ae14c0"} err="failed to get container status \"b7096418373c3bdc542d7806ab775777fc04f1e1597b6e98aa7a110bd1ae14c0\": rpc error: code = NotFound desc = could not find container \"b7096418373c3bdc542d7806ab775777fc04f1e1597b6e98aa7a110bd1ae14c0\": container with ID starting with b7096418373c3bdc542d7806ab775777fc04f1e1597b6e98aa7a110bd1ae14c0 not found: ID does not exist" Nov 03 17:16:04.090202 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.090175 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-credential-socket\") pod \"c844d192-95a9-4170-8af6-1b421fb533b5\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " Nov 03 17:16:04.090320 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.090221 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c844d192-95a9-4170-8af6-1b421fb533b5-kube-api-access-nprwg\" (UniqueName: \"kubernetes.io/projected/c844d192-95a9-4170-8af6-1b421fb533b5-kube-api-access-nprwg\") pod \"c844d192-95a9-4170-8af6-1b421fb533b5\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " Nov 03 17:16:04.090320 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.090242 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c844d192-95a9-4170-8af6-1b421fb533b5-istio-token\" (UniqueName: \"kubernetes.io/projected/c844d192-95a9-4170-8af6-1b421fb533b5-istio-token\") pod \"c844d192-95a9-4170-8af6-1b421fb533b5\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " Nov 03 17:16:04.090320 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.090261 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-workload-certs\") pod \"c844d192-95a9-4170-8af6-1b421fb533b5\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " Nov 03 17:16:04.090320 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.090308 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c844d192-95a9-4170-8af6-1b421fb533b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c844d192-95a9-4170-8af6-1b421fb533b5-istio-podinfo\") pod \"c844d192-95a9-4170-8af6-1b421fb533b5\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " Nov 03 17:16:04.090557 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.090368 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c844d192-95a9-4170-8af6-1b421fb533b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c844d192-95a9-4170-8af6-1b421fb533b5-istiod-ca-cert\") pod \"c844d192-95a9-4170-8af6-1b421fb533b5\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " Nov 03 17:16:04.090557 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.090433 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-credential-socket" pod "c844d192-95a9-4170-8af6-1b421fb533b5" (UID: "c844d192-95a9-4170-8af6-1b421fb533b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:16:04.090665 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.090570 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-workload-certs" pod "c844d192-95a9-4170-8af6-1b421fb533b5" (UID: "c844d192-95a9-4170-8af6-1b421fb533b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:16:04.091512 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.090812 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c844d192-95a9-4170-8af6-1b421fb533b5-istiod-ca-cert" pod "c844d192-95a9-4170-8af6-1b421fb533b5" (UID: "c844d192-95a9-4170-8af6-1b421fb533b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:16:04.091512 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.090817 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-istio-data\") pod \"c844d192-95a9-4170-8af6-1b421fb533b5\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " Nov 03 17:16:04.091512 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.090982 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-istio-data" pod "c844d192-95a9-4170-8af6-1b421fb533b5" (UID: "c844d192-95a9-4170-8af6-1b421fb533b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:16:04.091512 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.090923 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-istio-envoy\") pod \"c844d192-95a9-4170-8af6-1b421fb533b5\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " Nov 03 17:16:04.091512 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.091025 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-workload-socket\") pod \"c844d192-95a9-4170-8af6-1b421fb533b5\" (UID: \"c844d192-95a9-4170-8af6-1b421fb533b5\") " Nov 03 17:16:04.091512 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.091279 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-workload-socket" pod "c844d192-95a9-4170-8af6-1b421fb533b5" (UID: "c844d192-95a9-4170-8af6-1b421fb533b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:16:04.092144 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.091889 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:16:04.092144 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.091911 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:16:04.092144 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.091925 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c844d192-95a9-4170-8af6-1b421fb533b5-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:16:04.092144 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.091936 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:16:04.092144 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.091947 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:16:04.093002 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.092684 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c844d192-95a9-4170-8af6-1b421fb533b5-kube-api-access-nprwg" pod "c844d192-95a9-4170-8af6-1b421fb533b5" (UID: "c844d192-95a9-4170-8af6-1b421fb533b5"). InnerVolumeSpecName "kube-api-access-nprwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:16:04.093163 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.093143 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c844d192-95a9-4170-8af6-1b421fb533b5-istio-token" pod "c844d192-95a9-4170-8af6-1b421fb533b5" (UID: "c844d192-95a9-4170-8af6-1b421fb533b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:16:04.093233 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.093217 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c844d192-95a9-4170-8af6-1b421fb533b5-istio-podinfo" pod "c844d192-95a9-4170-8af6-1b421fb533b5" (UID: "c844d192-95a9-4170-8af6-1b421fb533b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:16:04.093522 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.093508 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-istio-envoy" pod "c844d192-95a9-4170-8af6-1b421fb533b5" (UID: "c844d192-95a9-4170-8af6-1b421fb533b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:16:04.193341 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.193312 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nprwg\" (UniqueName: \"kubernetes.io/projected/c844d192-95a9-4170-8af6-1b421fb533b5-kube-api-access-nprwg\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:16:04.193341 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.193339 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c844d192-95a9-4170-8af6-1b421fb533b5-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:16:04.193341 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.193348 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c844d192-95a9-4170-8af6-1b421fb533b5-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:16:04.193579 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.193356 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c844d192-95a9-4170-8af6-1b421fb533b5-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:16:04.253287 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.253254 4947 patch_prober.go:29] interesting pod/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 17:16:04.253958 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.253926 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" podUID="a852b387-ca23-4e7a-b793-788feb393176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 17:16:04.451764 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.451115 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw"] Nov 03 17:16:04.451764 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.451273 4947 patch_prober.go:29] interesting pod/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 03 17:16:04.451764 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.451331 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" podUID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 03 17:16:04.458785 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.458749 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-f5xzw"] Nov 03 17:16:04.569686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:04.569609 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c844d192-95a9-4170-8af6-1b421fb533b5" path="/var/lib/kubelet/pods/c844d192-95a9-4170-8af6-1b421fb533b5/volumes" Nov 03 17:16:05.253682 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:05.253648 4947 patch_prober.go:29] interesting pod/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 17:16:05.254249 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:05.253815 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" podUID="a852b387-ca23-4e7a-b793-788feb393176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 17:16:05.450897 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:05.450840 4947 patch_prober.go:29] interesting pod/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 03 17:16:05.451200 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:05.451170 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" podUID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 03 17:16:06.204868 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:06.204825 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:16:06.252888 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:06.252850 4947 patch_prober.go:29] interesting pod/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 17:16:06.253102 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:06.252901 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" podUID="a852b387-ca23-4e7a-b793-788feb393176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 17:16:06.450481 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:06.450441 4947 patch_prober.go:29] interesting pod/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" start-of-body= Nov 03 17:16:06.450957 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:06.450497 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" podUID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.241:15021/healthz/ready\": dial tcp 10.132.0.241:15021: connect: connection refused" Nov 03 17:16:07.253335 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:07.253296 4947 patch_prober.go:29] interesting pod/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 17:16:07.253628 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:07.253358 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" podUID="a852b387-ca23-4e7a-b793-788feb393176" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 17:16:07.458981 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:07.458937 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:16:07.470718 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:07.470044 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:16:07.475744 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:07.473566 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:16:08.258755 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:08.258723 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:16:08.260133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:08.260108 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:16:25.819282 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:25.819242 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:16:30.153450 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:30.153371 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:16:38.895853 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:38.895810 4947 patch_prober.go:29] interesting pod/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:16:38.896382 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:38.895875 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" podUID="c7a45866-e8e8-4731-b688-4674bf874a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:16:39.674465 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:39.674433 4947 generic.go:356] "Generic (PLEG): container finished" podID="c7a45866-e8e8-4731-b688-4674bf874a44" containerID="60ef2342f8d84e076a0b8bffa2ac96049cdd911af75d4f35580ee3e100d003aa" exitCode=0 Nov 03 17:16:39.674722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:39.674513 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" event={"ID":"c7a45866-e8e8-4731-b688-4674bf874a44","Type":"ContainerDied","Data":"60ef2342f8d84e076a0b8bffa2ac96049cdd911af75d4f35580ee3e100d003aa"} Nov 03 17:16:39.675058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:39.675038 4947 scope.go:122] "RemoveContainer" containerID="60ef2342f8d84e076a0b8bffa2ac96049cdd911af75d4f35580ee3e100d003aa" Nov 03 17:16:39.677085 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:39.677055 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:16:39.677181 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:39.677122 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:16:39.677181 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:39.677169 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:16:40.684319 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:40.684278 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" event={"ID":"c7a45866-e8e8-4731-b688-4674bf874a44","Type":"ContainerStarted","Data":"6da70a148d77c7926305be582fc27a381aee3d79064ef17403d98f1fcbc39f2f"} Nov 03 17:16:40.947825 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:40.947739 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:16:40.948769 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:40.948741 4947 patch_prober.go:29] interesting pod/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Nov 03 17:16:40.948853 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:40.948807 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" podUID="c7a45866-e8e8-4731-b688-4674bf874a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Nov 03 17:16:41.958152 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:41.958111 4947 patch_prober.go:29] interesting pod/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Nov 03 17:16:41.958795 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:41.958760 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" podUID="c7a45866-e8e8-4731-b688-4674bf874a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Nov 03 17:16:42.953775 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:42.953390 4947 patch_prober.go:29] interesting pod/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Nov 03 17:16:42.953775 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:42.953441 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" podUID="c7a45866-e8e8-4731-b688-4674bf874a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Nov 03 17:16:43.948312 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:43.948269 4947 patch_prober.go:29] interesting pod/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Nov 03 17:16:43.948722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:43.948336 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" podUID="c7a45866-e8e8-4731-b688-4674bf874a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Nov 03 17:16:44.948140 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:44.948100 4947 patch_prober.go:29] interesting pod/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Nov 03 17:16:44.948359 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:44.948161 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" podUID="c7a45866-e8e8-4731-b688-4674bf874a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Nov 03 17:16:45.948329 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:45.948285 4947 patch_prober.go:29] interesting pod/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Nov 03 17:16:45.948517 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:45.948364 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" podUID="c7a45866-e8e8-4731-b688-4674bf874a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Nov 03 17:16:46.948351 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:46.948314 4947 patch_prober.go:29] interesting pod/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body= Nov 03 17:16:46.948572 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:46.948380 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" podUID="c7a45866-e8e8-4731-b688-4674bf874a44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" Nov 03 17:16:47.953776 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:47.953739 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:16:47.954216 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:47.954197 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:16:47.955064 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:47.955030 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:16:48.202964 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.202927 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt"] Nov 03 17:16:48.204140 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.204070 4947 cpu_manager.go:414] "containerMap: removing container" podUID="c844d192-95a9-4170-8af6-1b421fb533b5" containerName="istio-proxy" Nov 03 17:16:48.204140 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.204095 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c844d192-95a9-4170-8af6-1b421fb533b5" containerName="istio-proxy" Nov 03 17:16:48.204336 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.204288 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="c844d192-95a9-4170-8af6-1b421fb533b5" containerName="istio-proxy" Nov 03 17:16:48.208338 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.208318 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" Nov 03 17:16:48.226132 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.226098 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-9c4762-dockercfg-5sp79\"" Nov 03 17:16:48.226309 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.226195 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-9c4762\"" Nov 03 17:16:48.231604 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.231577 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt"] Nov 03 17:16:48.695992 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.695950 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kknlz\" (UniqueName: \"kubernetes.io/projected/b2938160-d944-4c19-abd6-f634b57b920b-kube-api-access-kknlz\") pod \"payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt\" (UID: \"b2938160-d944-4c19-abd6-f634b57b920b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" Nov 03 17:16:48.696242 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.696221 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2938160-d944-4c19-abd6-f634b57b920b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt\" (UID: \"b2938160-d944-4c19-abd6-f634b57b920b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" Nov 03 17:16:48.797612 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.797572 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kknlz\" (UniqueName: \"kubernetes.io/projected/b2938160-d944-4c19-abd6-f634b57b920b-kube-api-access-kknlz\") pod \"payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt\" (UID: \"b2938160-d944-4c19-abd6-f634b57b920b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" Nov 03 17:16:48.798943 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.797779 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2938160-d944-4c19-abd6-f634b57b920b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt\" (UID: \"b2938160-d944-4c19-abd6-f634b57b920b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" Nov 03 17:16:48.798943 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.798778 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2938160-d944-4c19-abd6-f634b57b920b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt\" (UID: \"b2938160-d944-4c19-abd6-f634b57b920b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" Nov 03 17:16:48.823510 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.823480 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kknlz\" (UniqueName: \"kubernetes.io/projected/b2938160-d944-4c19-abd6-f634b57b920b-kube-api-access-kknlz\") pod \"payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt\" (UID: \"b2938160-d944-4c19-abd6-f634b57b920b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" Nov 03 17:16:48.845495 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.845464 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" Nov 03 17:16:48.925788 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.924063 4947 generic.go:356] "Generic (PLEG): container finished" podID="1508d21f-1858-4232-be82-bcda731862b9" containerID="03f58fdf817c9e0ca7390c901b4a4953c998c940c379b158ad2c2a1ea80e9245" exitCode=0 Nov 03 17:16:48.925788 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.925201 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" event={"ID":"1508d21f-1858-4232-be82-bcda731862b9","Type":"ContainerDied","Data":"03f58fdf817c9e0ca7390c901b4a4953c998c940c379b158ad2c2a1ea80e9245"} Nov 03 17:16:48.925788 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.925619 4947 scope.go:122] "RemoveContainer" containerID="03f58fdf817c9e0ca7390c901b4a4953c998c940c379b158ad2c2a1ea80e9245" Nov 03 17:16:48.930272 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.930203 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:16:48.930402 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.930299 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:16:48.930402 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:48.930348 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:16:49.160577 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:16:49.160544 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2938160_d944_4c19_abd6_f634b57b920b.slice/crio-ab4f3026e296b7e95d8f7d57716bd46d3c889668e6be638d9d1f93a3b09ea9cc WatchSource:0}: Error finding container ab4f3026e296b7e95d8f7d57716bd46d3c889668e6be638d9d1f93a3b09ea9cc: Status 404 returned error can't find the container with id ab4f3026e296b7e95d8f7d57716bd46d3c889668e6be638d9d1f93a3b09ea9cc Nov 03 17:16:49.162377 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:49.162354 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt"] Nov 03 17:16:49.936359 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:49.936325 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" event={"ID":"b2938160-d944-4c19-abd6-f634b57b920b","Type":"ContainerStarted","Data":"291e76ab329beebdc76ff9a1ac2d0461bac5383b9878ac36b66a0ce99e822047"} Nov 03 17:16:49.936654 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:49.936634 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" event={"ID":"b2938160-d944-4c19-abd6-f634b57b920b","Type":"ContainerStarted","Data":"ab4f3026e296b7e95d8f7d57716bd46d3c889668e6be638d9d1f93a3b09ea9cc"} Nov 03 17:16:49.939954 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:49.939930 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" Nov 03 17:16:49.944908 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:49.944851 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" event={"ID":"1508d21f-1858-4232-be82-bcda731862b9","Type":"ContainerStarted","Data":"0e75f3ace41c88506258a0b364ffc210f286edb0f4a52affa3d6112078255f04"} Nov 03 17:16:49.995204 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:49.995156 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" podStartSLOduration=1.9951429200000002 podStartE2EDuration="1.99514292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:49.991892343 +0000 UTC m=+3753.580892310" watchObservedRunningTime="2026-11-03 17:16:49.99514292 +0000 UTC m=+3753.584142881" Nov 03 17:16:50.846547 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:50.846511 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:16:50.848183 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:50.848158 4947 patch_prober.go:29] interesting pod/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Nov 03 17:16:50.848310 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:50.848205 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" podUID="1508d21f-1858-4232-be82-bcda731862b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Nov 03 17:16:51.846722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:51.846670 4947 patch_prober.go:29] interesting pod/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Nov 03 17:16:51.847134 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:51.846752 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" podUID="1508d21f-1858-4232-be82-bcda731862b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Nov 03 17:16:52.846385 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:52.846347 4947 patch_prober.go:29] interesting pod/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Nov 03 17:16:52.846581 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:52.846408 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" podUID="1508d21f-1858-4232-be82-bcda731862b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Nov 03 17:16:53.439093 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:53.439056 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:16:53.850753 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:53.850716 4947 patch_prober.go:29] interesting pod/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Nov 03 17:16:53.851058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:53.851023 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" podUID="1508d21f-1858-4232-be82-bcda731862b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Nov 03 17:16:54.195725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.195637 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq"] Nov 03 17:16:54.204399 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.204373 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" Nov 03 17:16:54.230184 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.230149 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq"] Nov 03 17:16:54.233670 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.233632 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"kube-root-ca.crt\"" Nov 03 17:16:54.233841 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.233804 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"openshift-service-ca.crt\"" Nov 03 17:16:54.233841 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.233814 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e0e63d\"" Nov 03 17:16:54.233841 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.233659 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"routing-overlay\"" Nov 03 17:16:54.241300 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.241279 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"agc-cred-22a5b1e7c2da\"" Nov 03 17:16:54.241431 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.241399 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e0e63d\"/\"payload-processing-external-model-e2e--2172597b-dockercfg-xrxj5\"" Nov 03 17:16:54.346805 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.346770 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq\" (UID: \"f40f5832-fdbb-42bb-8849-37aad769d61b\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" Nov 03 17:16:54.347001 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.346813 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq\" (UID: \"f40f5832-fdbb-42bb-8849-37aad769d61b\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" Nov 03 17:16:54.347001 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.346965 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqff2\" (UniqueName: \"kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-kube-api-access-sqff2\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq\" (UID: \"f40f5832-fdbb-42bb-8849-37aad769d61b\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" Nov 03 17:16:54.347001 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.346988 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq\" (UID: \"f40f5832-fdbb-42bb-8849-37aad769d61b\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" Nov 03 17:16:54.449192 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.449088 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq\" (UID: \"f40f5832-fdbb-42bb-8849-37aad769d61b\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" Nov 03 17:16:54.449192 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.449150 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq\" (UID: \"f40f5832-fdbb-42bb-8849-37aad769d61b\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" Nov 03 17:16:54.449824 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.449295 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqff2\" (UniqueName: \"kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-kube-api-access-sqff2\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq\" (UID: \"f40f5832-fdbb-42bb-8849-37aad769d61b\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" Nov 03 17:16:54.449824 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.449334 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq\" (UID: \"f40f5832-fdbb-42bb-8849-37aad769d61b\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" Nov 03 17:16:54.450212 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.450185 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq\" (UID: \"f40f5832-fdbb-42bb-8849-37aad769d61b\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" Nov 03 17:16:54.450341 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.450319 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq\" (UID: \"f40f5832-fdbb-42bb-8849-37aad769d61b\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" Nov 03 17:16:54.452165 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.452139 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq\" (UID: \"f40f5832-fdbb-42bb-8849-37aad769d61b\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" Nov 03 17:16:54.474153 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.474121 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqff2\" (UniqueName: \"kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-kube-api-access-sqff2\") pod \"payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq\" (UID: \"f40f5832-fdbb-42bb-8849-37aad769d61b\") " pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" Nov 03 17:16:54.543668 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.543298 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" Nov 03 17:16:54.853289 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.853249 4947 patch_prober.go:29] interesting pod/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Nov 03 17:16:54.853650 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:54.853603 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" podUID="1508d21f-1858-4232-be82-bcda731862b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Nov 03 17:16:55.462940 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:55.462900 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq"] Nov 03 17:16:55.518868 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:16:55.518840 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40f5832_fdbb_42bb_8849_37aad769d61b.slice/crio-bea9220bd35d9a2c3967e5cd24ce57327c14196f9c0b5dbbbde395ea7a8b75ee WatchSource:0}: Error finding container bea9220bd35d9a2c3967e5cd24ce57327c14196f9c0b5dbbbde395ea7a8b75ee: Status 404 returned error can't find the container with id bea9220bd35d9a2c3967e5cd24ce57327c14196f9c0b5dbbbde395ea7a8b75ee Nov 03 17:16:55.847755 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:55.847710 4947 patch_prober.go:29] interesting pod/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" start-of-body= Nov 03 17:16:55.847999 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:55.847782 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" podUID="1508d21f-1858-4232-be82-bcda731862b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.242:15021/healthz/ready\": dial tcp 10.132.0.242:15021: connect: connection refused" Nov 03 17:16:56.099916 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:56.099826 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" event={"ID":"f40f5832-fdbb-42bb-8849-37aad769d61b","Type":"ContainerStarted","Data":"5a3356379f2dc1fad77f134b3dcff4414f6c85adf9af9027cab2044571088c73"} Nov 03 17:16:56.099916 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:56.099886 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" event={"ID":"f40f5832-fdbb-42bb-8849-37aad769d61b","Type":"ContainerStarted","Data":"bea9220bd35d9a2c3967e5cd24ce57327c14196f9c0b5dbbbde395ea7a8b75ee"} Nov 03 17:16:56.099916 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:56.099912 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" Nov 03 17:16:56.147638 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:56.147582 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" podStartSLOduration=2.147569349 podStartE2EDuration="2.147569349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:16:56.145734213 +0000 UTC m=+3759.734734183" watchObservedRunningTime="2026-11-03 17:16:56.147569349 +0000 UTC m=+3759.736569315" Nov 03 17:16:56.150501 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:56.150472 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e0e63d/agc-cred-22a5b1e7c2da: secret "agc-cred-22a5b1e7c2da" not found Nov 03 17:16:56.150501 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:56.150495 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq: secret "agc-cred-22a5b1e7c2da" not found Nov 03 17:16:56.150739 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:56.150540 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-provider-credentials podName:f40f5832-fdbb-42bb-8849-37aad769d61b nodeName:}" failed. No retries permitted until 2026-11-03 17:16:56.650527125 +0000 UTC m=+3760.239527072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" (UID: "f40f5832-fdbb-42bb-8849-37aad769d61b") : secret "agc-cred-22a5b1e7c2da" not found Nov 03 17:16:56.666746 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:56.666714 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e0e63d/agc-cred-22a5b1e7c2da: secret "agc-cred-22a5b1e7c2da" not found Nov 03 17:16:56.666746 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:56.666741 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq: secret "agc-cred-22a5b1e7c2da" not found Nov 03 17:16:56.667235 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:56.666813 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-provider-credentials podName:f40f5832-fdbb-42bb-8849-37aad769d61b nodeName:}" failed. No retries permitted until 2026-11-03 17:16:57.666797627 +0000 UTC m=+3761.255797574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" (UID: "f40f5832-fdbb-42bb-8849-37aad769d61b") : secret "agc-cred-22a5b1e7c2da" not found Nov 03 17:16:56.851649 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:56.851617 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:16:56.852924 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:56.852902 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:16:56.989804 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:56.989187 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq"] Nov 03 17:16:57.108293 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:16:57.108247 4947 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" secret="" err="secret \"payload-processing-external-model-e2e--2172597b-dockercfg-xrxj5\" not found" Nov 03 17:16:57.176520 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:57.176490 4947 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-plugins-e2e-extmod-b-e0e63d: configmap "payload-processing-external-model-plugins-e2e-extmod-b-e0e63d" not found Nov 03 17:16:57.176757 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:57.176564 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-plugins-config-volume podName:f40f5832-fdbb-42bb-8849-37aad769d61b nodeName:}" failed. No retries permitted until 2026-11-03 17:16:57.676546071 +0000 UTC m=+3761.265546041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" (UID: "f40f5832-fdbb-42bb-8849-37aad769d61b") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-e0e63d" not found Nov 03 17:16:57.693244 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:57.693213 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e0e63d/agc-cred-22a5b1e7c2da: secret "agc-cred-22a5b1e7c2da" not found Nov 03 17:16:57.693832 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:57.693815 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq: secret "agc-cred-22a5b1e7c2da" not found Nov 03 17:16:57.694002 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:57.693991 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-provider-credentials podName:f40f5832-fdbb-42bb-8849-37aad769d61b nodeName:}" failed. No retries permitted until 2026-11-03 17:16:59.6939702 +0000 UTC m=+3763.282970151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" (UID: "f40f5832-fdbb-42bb-8849-37aad769d61b") : secret "agc-cred-22a5b1e7c2da" not found Nov 03 17:16:57.694389 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:57.693786 4947 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-plugins-e2e-extmod-b-e0e63d: configmap "payload-processing-external-model-plugins-e2e-extmod-b-e0e63d" not found Nov 03 17:16:57.694531 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:57.694520 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-plugins-config-volume podName:f40f5832-fdbb-42bb-8849-37aad769d61b nodeName:}" failed. No retries permitted until 2026-11-03 17:16:58.694504345 +0000 UTC m=+3762.283504295 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" (UID: "f40f5832-fdbb-42bb-8849-37aad769d61b") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-e0e63d" not found Nov 03 17:16:58.728485 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:58.728446 4947 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-plugins-e2e-extmod-b-e0e63d: configmap "payload-processing-external-model-plugins-e2e-extmod-b-e0e63d" not found Nov 03 17:16:58.729430 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:58.729404 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-plugins-config-volume podName:f40f5832-fdbb-42bb-8849-37aad769d61b nodeName:}" failed. No retries permitted until 2026-11-03 17:17:00.729376468 +0000 UTC m=+3764.318376433 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" (UID: "f40f5832-fdbb-42bb-8849-37aad769d61b") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-e0e63d" not found Nov 03 17:16:59.756095 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:59.756061 4947 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-e0e63d/agc-cred-22a5b1e7c2da: secret "agc-cred-22a5b1e7c2da" not found Nov 03 17:16:59.756095 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:59.756088 4947 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq: secret "agc-cred-22a5b1e7c2da" not found Nov 03 17:16:59.756553 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:16:59.756146 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-provider-credentials podName:f40f5832-fdbb-42bb-8849-37aad769d61b nodeName:}" failed. No retries permitted until 2026-11-03 17:17:03.756129084 +0000 UTC m=+3767.345129031 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" (UID: "f40f5832-fdbb-42bb-8849-37aad769d61b") : secret "agc-cred-22a5b1e7c2da" not found Nov 03 17:17:00.132400 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:17:00.132316 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:17:00.774509 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:17:00.774472 4947 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-plugins-e2e-extmod-b-e0e63d: configmap "payload-processing-external-model-plugins-e2e-extmod-b-e0e63d" not found Nov 03 17:17:00.775037 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:17:00.774556 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-plugins-config-volume podName:f40f5832-fdbb-42bb-8849-37aad769d61b nodeName:}" failed. No retries permitted until 2026-11-03 17:17:04.77453792 +0000 UTC m=+3768.363537867 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" (UID: "f40f5832-fdbb-42bb-8849-37aad769d61b") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-e0e63d" not found Nov 03 17:17:02.030091 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:02.030059 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" Nov 03 17:17:03.118647 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.118599 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" podUID="f40f5832-fdbb-42bb-8849-37aad769d61b" containerName="payload-processing" containerID="cri-o://5a3356379f2dc1fad77f134b3dcff4414f6c85adf9af9027cab2044571088c73" gracePeriod=25 Nov 03 17:17:03.554248 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.554220 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" Nov 03 17:17:03.631406 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.631373 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-kube-api-access-sqff2\" (UniqueName: \"kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-kube-api-access-sqff2\") pod \"f40f5832-fdbb-42bb-8849-37aad769d61b\" (UID: \"f40f5832-fdbb-42bb-8849-37aad769d61b\") " Nov 03 17:17:03.631593 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.631446 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-routing-overlay-volume\") pod \"f40f5832-fdbb-42bb-8849-37aad769d61b\" (UID: \"f40f5832-fdbb-42bb-8849-37aad769d61b\") " Nov 03 17:17:03.631683 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.631668 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-plugins-config-volume\") pod \"f40f5832-fdbb-42bb-8849-37aad769d61b\" (UID: \"f40f5832-fdbb-42bb-8849-37aad769d61b\") " Nov 03 17:17:03.632133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.631831 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-provider-credentials\") pod \"f40f5832-fdbb-42bb-8849-37aad769d61b\" (UID: \"f40f5832-fdbb-42bb-8849-37aad769d61b\") " Nov 03 17:17:03.632133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.632085 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-routing-overlay-volume" pod "f40f5832-fdbb-42bb-8849-37aad769d61b" (UID: "f40f5832-fdbb-42bb-8849-37aad769d61b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:17:03.633818 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.633782 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-plugins-config-volume" pod "f40f5832-fdbb-42bb-8849-37aad769d61b" (UID: "f40f5832-fdbb-42bb-8849-37aad769d61b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:17:03.636470 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.634309 4947 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-routing-overlay-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:17:03.636470 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.634335 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40f5832-fdbb-42bb-8849-37aad769d61b-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:17:03.636470 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.634919 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-provider-credentials" pod "f40f5832-fdbb-42bb-8849-37aad769d61b" (UID: "f40f5832-fdbb-42bb-8849-37aad769d61b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:17:03.636470 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.635636 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-kube-api-access-sqff2" pod "f40f5832-fdbb-42bb-8849-37aad769d61b" (UID: "f40f5832-fdbb-42bb-8849-37aad769d61b"). InnerVolumeSpecName "kube-api-access-sqff2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:17:03.736295 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.736221 4947 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-provider-credentials\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:17:03.736295 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.736247 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqff2\" (UniqueName: \"kubernetes.io/projected/f40f5832-fdbb-42bb-8849-37aad769d61b-kube-api-access-sqff2\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:17:03.808280 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.808249 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9"] Nov 03 17:17:03.809105 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.809086 4947 cpu_manager.go:414] "containerMap: removing container" podUID="f40f5832-fdbb-42bb-8849-37aad769d61b" containerName="payload-processing" Nov 03 17:17:03.809105 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.809105 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40f5832-fdbb-42bb-8849-37aad769d61b" containerName="payload-processing" Nov 03 17:17:03.809300 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.809289 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40f5832-fdbb-42bb-8849-37aad769d61b" containerName="payload-processing" Nov 03 17:17:03.812235 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.812217 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" Nov 03 17:17:03.832756 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.830592 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-547925\"" Nov 03 17:17:03.833855 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.833601 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-547925-dockercfg-hxlrd\"" Nov 03 17:17:03.838798 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.838771 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9"] Nov 03 17:17:03.967722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.962450 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407a82a9-162d-4120-82a1-f6022279dced-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9\" (UID: \"407a82a9-162d-4120-82a1-f6022279dced\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" Nov 03 17:17:03.967722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:03.962755 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c65hk\" (UniqueName: \"kubernetes.io/projected/407a82a9-162d-4120-82a1-f6022279dced-kube-api-access-c65hk\") pod \"payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9\" (UID: \"407a82a9-162d-4120-82a1-f6022279dced\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" Nov 03 17:17:04.129732 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:04.126381 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c65hk\" (UniqueName: \"kubernetes.io/projected/407a82a9-162d-4120-82a1-f6022279dced-kube-api-access-c65hk\") pod \"payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9\" (UID: \"407a82a9-162d-4120-82a1-f6022279dced\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" Nov 03 17:17:04.129732 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:04.126759 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407a82a9-162d-4120-82a1-f6022279dced-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9\" (UID: \"407a82a9-162d-4120-82a1-f6022279dced\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" Nov 03 17:17:04.129732 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:04.127840 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407a82a9-162d-4120-82a1-f6022279dced-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9\" (UID: \"407a82a9-162d-4120-82a1-f6022279dced\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" Nov 03 17:17:04.244049 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:04.244005 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c65hk\" (UniqueName: \"kubernetes.io/projected/407a82a9-162d-4120-82a1-f6022279dced-kube-api-access-c65hk\") pod \"payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9\" (UID: \"407a82a9-162d-4120-82a1-f6022279dced\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" Nov 03 17:17:04.247540 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:04.247495 4947 generic.go:356] "Generic (PLEG): container finished" podID="f40f5832-fdbb-42bb-8849-37aad769d61b" containerID="5a3356379f2dc1fad77f134b3dcff4414f6c85adf9af9027cab2044571088c73" exitCode=0 Nov 03 17:17:04.248687 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:04.247839 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" Nov 03 17:17:04.248687 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:04.247901 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" event={"ID":"f40f5832-fdbb-42bb-8849-37aad769d61b","Type":"ContainerDied","Data":"5a3356379f2dc1fad77f134b3dcff4414f6c85adf9af9027cab2044571088c73"} Nov 03 17:17:04.248687 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:04.247956 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq" event={"ID":"f40f5832-fdbb-42bb-8849-37aad769d61b","Type":"ContainerDied","Data":"bea9220bd35d9a2c3967e5cd24ce57327c14196f9c0b5dbbbde395ea7a8b75ee"} Nov 03 17:17:04.248687 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:04.248000 4947 scope.go:122] "RemoveContainer" containerID="5a3356379f2dc1fad77f134b3dcff4414f6c85adf9af9027cab2044571088c73" Nov 03 17:17:04.270552 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:04.270522 4947 scope.go:122] "RemoveContainer" containerID="5a3356379f2dc1fad77f134b3dcff4414f6c85adf9af9027cab2044571088c73" Nov 03 17:17:04.271096 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:17:04.271066 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a3356379f2dc1fad77f134b3dcff4414f6c85adf9af9027cab2044571088c73\": container with ID starting with 5a3356379f2dc1fad77f134b3dcff4414f6c85adf9af9027cab2044571088c73 not found: ID does not exist" containerID="5a3356379f2dc1fad77f134b3dcff4414f6c85adf9af9027cab2044571088c73" Nov 03 17:17:04.271195 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:04.271111 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a3356379f2dc1fad77f134b3dcff4414f6c85adf9af9027cab2044571088c73"} err="failed to get container status \"5a3356379f2dc1fad77f134b3dcff4414f6c85adf9af9027cab2044571088c73\": rpc error: code = NotFound desc = could not find container \"5a3356379f2dc1fad77f134b3dcff4414f6c85adf9af9027cab2044571088c73\": container with ID starting with 5a3356379f2dc1fad77f134b3dcff4414f6c85adf9af9027cab2044571088c73 not found: ID does not exist" Nov 03 17:17:04.310190 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:04.310156 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq"] Nov 03 17:17:04.314261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:04.314232 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e0e63d/payload-processing-external-model-e2e-extmod-b-e0e63d-d58crslvq"] Nov 03 17:17:04.427048 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:04.426964 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" Nov 03 17:17:04.570155 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:04.570111 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40f5832-fdbb-42bb-8849-37aad769d61b" path="/var/lib/kubelet/pods/f40f5832-fdbb-42bb-8849-37aad769d61b/volumes" Nov 03 17:17:04.812128 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:04.812089 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9"] Nov 03 17:17:04.812504 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:17:04.812479 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407a82a9_162d_4120_82a1_f6022279dced.slice/crio-06a1a1238f7c23048a604aab1649927b9b15c744562ed9baebc7ef24fd9be360 WatchSource:0}: Error finding container 06a1a1238f7c23048a604aab1649927b9b15c744562ed9baebc7ef24fd9be360: Status 404 returned error can't find the container with id 06a1a1238f7c23048a604aab1649927b9b15c744562ed9baebc7ef24fd9be360 Nov 03 17:17:05.310246 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:05.310193 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" event={"ID":"407a82a9-162d-4120-82a1-f6022279dced","Type":"ContainerStarted","Data":"a82226526df3d3f91339630ab15d1253141d3f36c17b4a5d62d40855416c9c0d"} Nov 03 17:17:05.310830 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:05.310269 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" event={"ID":"407a82a9-162d-4120-82a1-f6022279dced","Type":"ContainerStarted","Data":"06a1a1238f7c23048a604aab1649927b9b15c744562ed9baebc7ef24fd9be360"} Nov 03 17:17:05.311526 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:05.311500 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" Nov 03 17:17:11.968429 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:11.968362 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" podStartSLOduration=8.96834393 podStartE2EDuration="8.96834393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:05.376268784 +0000 UTC m=+3768.965268753" watchObservedRunningTime="2026-11-03 17:17:11.96834393 +0000 UTC m=+3775.557343949" Nov 03 17:17:11.971152 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:11.971129 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg"] Nov 03 17:17:11.976786 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:11.976766 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" Nov 03 17:17:12.005487 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.005458 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg"] Nov 03 17:17:12.006824 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.006779 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-472bee\"" Nov 03 17:17:12.006958 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.006859 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-472bee-dockercfg-7hckr\"" Nov 03 17:17:12.038195 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.030125 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2"] Nov 03 17:17:12.085721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.082390 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" Nov 03 17:17:12.342012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.341967 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jvn2\" (UniqueName: \"kubernetes.io/projected/baec733a-0e6d-44c7-8807-4f7437806318-kube-api-access-2jvn2\") pod \"payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg\" (UID: \"baec733a-0e6d-44c7-8807-4f7437806318\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" Nov 03 17:17:12.348371 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.347899 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baec733a-0e6d-44c7-8807-4f7437806318-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg\" (UID: \"baec733a-0e6d-44c7-8807-4f7437806318\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" Nov 03 17:17:12.443124 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.443089 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2"] Nov 03 17:17:12.452822 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.452788 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98e92c6-7bc2-4211-b7c2-524f7eddfe88-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2\" (UID: \"f98e92c6-7bc2-4211-b7c2-524f7eddfe88\") " pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" Nov 03 17:17:12.453075 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.453057 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvn2\" (UniqueName: \"kubernetes.io/projected/baec733a-0e6d-44c7-8807-4f7437806318-kube-api-access-2jvn2\") pod \"payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg\" (UID: \"baec733a-0e6d-44c7-8807-4f7437806318\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" Nov 03 17:17:12.453137 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.453102 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baec733a-0e6d-44c7-8807-4f7437806318-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg\" (UID: \"baec733a-0e6d-44c7-8807-4f7437806318\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" Nov 03 17:17:12.453196 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.453132 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv5gr\" (UniqueName: \"kubernetes.io/projected/f98e92c6-7bc2-4211-b7c2-524f7eddfe88-kube-api-access-tv5gr\") pod \"payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2\" (UID: \"f98e92c6-7bc2-4211-b7c2-524f7eddfe88\") " pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" Nov 03 17:17:12.454530 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.454503 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baec733a-0e6d-44c7-8807-4f7437806318-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg\" (UID: \"baec733a-0e6d-44c7-8807-4f7437806318\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" Nov 03 17:17:12.478855 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.478826 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvn2\" (UniqueName: \"kubernetes.io/projected/baec733a-0e6d-44c7-8807-4f7437806318-kube-api-access-2jvn2\") pod \"payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg\" (UID: \"baec733a-0e6d-44c7-8807-4f7437806318\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" Nov 03 17:17:12.554475 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.554425 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv5gr\" (UniqueName: \"kubernetes.io/projected/f98e92c6-7bc2-4211-b7c2-524f7eddfe88-kube-api-access-tv5gr\") pod \"payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2\" (UID: \"f98e92c6-7bc2-4211-b7c2-524f7eddfe88\") " pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" Nov 03 17:17:12.554664 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.554643 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98e92c6-7bc2-4211-b7c2-524f7eddfe88-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2\" (UID: \"f98e92c6-7bc2-4211-b7c2-524f7eddfe88\") " pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" Nov 03 17:17:12.555350 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.555332 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98e92c6-7bc2-4211-b7c2-524f7eddfe88-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2\" (UID: \"f98e92c6-7bc2-4211-b7c2-524f7eddfe88\") " pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" Nov 03 17:17:12.579056 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.579030 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv5gr\" (UniqueName: \"kubernetes.io/projected/f98e92c6-7bc2-4211-b7c2-524f7eddfe88-kube-api-access-tv5gr\") pod \"payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2\" (UID: \"f98e92c6-7bc2-4211-b7c2-524f7eddfe88\") " pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" Nov 03 17:17:12.591219 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.591193 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" Nov 03 17:17:12.757817 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.757786 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" Nov 03 17:17:12.908806 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:17:12.906848 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaec733a_0e6d_44c7_8807_4f7437806318.slice/crio-5de5562ca1aad63dac947eb9203d2378f247b821240b28279529876871c7b946 WatchSource:0}: Error finding container 5de5562ca1aad63dac947eb9203d2378f247b821240b28279529876871c7b946: Status 404 returned error can't find the container with id 5de5562ca1aad63dac947eb9203d2378f247b821240b28279529876871c7b946 Nov 03 17:17:12.915618 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:12.915590 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg"] Nov 03 17:17:13.107074 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:17:13.107027 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98e92c6_7bc2_4211_b7c2_524f7eddfe88.slice/crio-4778d912408aa913c5a762d8d3713388d8ea483c213bcd81f845f78ed0aa935f WatchSource:0}: Error finding container 4778d912408aa913c5a762d8d3713388d8ea483c213bcd81f845f78ed0aa935f: Status 404 returned error can't find the container with id 4778d912408aa913c5a762d8d3713388d8ea483c213bcd81f845f78ed0aa935f Nov 03 17:17:13.107496 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:13.107342 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2"] Nov 03 17:17:13.450436 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:13.450352 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" event={"ID":"baec733a-0e6d-44c7-8807-4f7437806318","Type":"ContainerStarted","Data":"f2a7114627a977bc158ef9c5bde1ae099a65355dd162523f3f9a260a7b3af98e"} Nov 03 17:17:13.450436 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:13.450408 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" event={"ID":"baec733a-0e6d-44c7-8807-4f7437806318","Type":"ContainerStarted","Data":"5de5562ca1aad63dac947eb9203d2378f247b821240b28279529876871c7b946"} Nov 03 17:17:13.450436 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:13.450428 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" Nov 03 17:17:13.452124 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:13.452099 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" event={"ID":"f98e92c6-7bc2-4211-b7c2-524f7eddfe88","Type":"ContainerStarted","Data":"2171c81f0ecd1fbcc9c3c91f6a830dc58d5227a4397a112642096344e14596fc"} Nov 03 17:17:13.452247 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:13.452140 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" event={"ID":"f98e92c6-7bc2-4211-b7c2-524f7eddfe88","Type":"ContainerStarted","Data":"4778d912408aa913c5a762d8d3713388d8ea483c213bcd81f845f78ed0aa935f"} Nov 03 17:17:13.452247 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:13.452227 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" Nov 03 17:17:13.499231 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:13.499183 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" podStartSLOduration=2.499169904 podStartE2EDuration="2.499169904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:13.494771167 +0000 UTC m=+3777.083771145" watchObservedRunningTime="2026-11-03 17:17:13.499169904 +0000 UTC m=+3777.088169872" Nov 03 17:17:13.542568 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:13.542515 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" podStartSLOduration=2.5425020419999997 podStartE2EDuration="2.542502042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:13.53847312 +0000 UTC m=+3777.127473088" watchObservedRunningTime="2026-11-03 17:17:13.542502042 +0000 UTC m=+3777.131502009" Nov 03 17:17:17.346840 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:17.346805 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" Nov 03 17:17:20.117375 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:20.117342 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz"] Nov 03 17:17:20.126358 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:20.126324 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" Nov 03 17:17:20.146734 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:20.146003 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz"] Nov 03 17:17:20.153330 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:20.153294 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-527051-serving-cert\"" Nov 03 17:17:20.194598 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:20.194565 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e32622d-7ba0-45ad-a866-05089e15f14e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz\" (UID: \"1e32622d-7ba0-45ad-a866-05089e15f14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" Nov 03 17:17:20.194781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:20.194755 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e32622d-7ba0-45ad-a866-05089e15f14e-maas-api-tls\") pod \"maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz\" (UID: \"1e32622d-7ba0-45ad-a866-05089e15f14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" Nov 03 17:17:20.194915 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:20.194896 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ttkh\" (UniqueName: \"kubernetes.io/projected/1e32622d-7ba0-45ad-a866-05089e15f14e-kube-api-access-6ttkh\") pod \"maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz\" (UID: \"1e32622d-7ba0-45ad-a866-05089e15f14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" Nov 03 17:17:20.195070 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:20.195055 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e32622d-7ba0-45ad-a866-05089e15f14e-metrics-certs\") pod \"maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz\" (UID: \"1e32622d-7ba0-45ad-a866-05089e15f14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" Nov 03 17:17:20.296257 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:20.296224 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e32622d-7ba0-45ad-a866-05089e15f14e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz\" (UID: \"1e32622d-7ba0-45ad-a866-05089e15f14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" Nov 03 17:17:20.296434 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:20.296310 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e32622d-7ba0-45ad-a866-05089e15f14e-maas-api-tls\") pod \"maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz\" (UID: \"1e32622d-7ba0-45ad-a866-05089e15f14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" Nov 03 17:17:20.296434 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:20.296373 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ttkh\" (UniqueName: \"kubernetes.io/projected/1e32622d-7ba0-45ad-a866-05089e15f14e-kube-api-access-6ttkh\") pod \"maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz\" (UID: \"1e32622d-7ba0-45ad-a866-05089e15f14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" Nov 03 17:17:20.296561 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:20.296484 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e32622d-7ba0-45ad-a866-05089e15f14e-metrics-certs\") pod \"maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz\" (UID: \"1e32622d-7ba0-45ad-a866-05089e15f14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" Nov 03 17:17:20.296968 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:20.296940 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e32622d-7ba0-45ad-a866-05089e15f14e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz\" (UID: \"1e32622d-7ba0-45ad-a866-05089e15f14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" Nov 03 17:17:20.298893 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:20.298871 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e32622d-7ba0-45ad-a866-05089e15f14e-maas-api-tls\") pod \"maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz\" (UID: \"1e32622d-7ba0-45ad-a866-05089e15f14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" Nov 03 17:17:20.299005 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:20.298976 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e32622d-7ba0-45ad-a866-05089e15f14e-metrics-certs\") pod \"maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz\" (UID: \"1e32622d-7ba0-45ad-a866-05089e15f14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" Nov 03 17:17:20.321488 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:20.321466 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ttkh\" (UniqueName: \"kubernetes.io/projected/1e32622d-7ba0-45ad-a866-05089e15f14e-kube-api-access-6ttkh\") pod \"maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz\" (UID: \"1e32622d-7ba0-45ad-a866-05089e15f14e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" Nov 03 17:17:20.465087 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:20.465005 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" Nov 03 17:17:20.774440 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:17:20.774407 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e32622d_7ba0_45ad_a866_05089e15f14e.slice/crio-a99ad3b761b78191e6ce4ba56ab06322d275519d5204cbb569ef0fde326b0f59 WatchSource:0}: Error finding container a99ad3b761b78191e6ce4ba56ab06322d275519d5204cbb569ef0fde326b0f59: Status 404 returned error can't find the container with id a99ad3b761b78191e6ce4ba56ab06322d275519d5204cbb569ef0fde326b0f59 Nov 03 17:17:20.775662 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:20.775637 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz"] Nov 03 17:17:21.590635 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:21.590222 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" event={"ID":"1e32622d-7ba0-45ad-a866-05089e15f14e","Type":"ContainerStarted","Data":"a99ad3b761b78191e6ce4ba56ab06322d275519d5204cbb569ef0fde326b0f59"} Nov 03 17:17:22.599878 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:22.599824 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" event={"ID":"1e32622d-7ba0-45ad-a866-05089e15f14e","Type":"ContainerStarted","Data":"b9b3e12c43d7301555a8cec805a39b21faba309a861c175fd462b77b82048cd0"} Nov 03 17:17:22.600331 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:22.599896 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" Nov 03 17:17:22.648573 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:22.648520 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" podStartSLOduration=1.89921453 podStartE2EDuration="2.648504238s" totalImagesPullingTime="749.289708ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:22.645652107 +0000 UTC m=+3786.234652075" watchObservedRunningTime="2026-11-03 17:17:22.648504238 +0000 UTC m=+3786.237504229" Nov 03 17:17:24.462896 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:24.462860 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" Nov 03 17:17:24.463341 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:24.462942 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" Nov 03 17:17:26.645423 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:26.645392 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:17:28.619858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:28.619825 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" Nov 03 17:17:30.131764 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:17:30.131725 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:17:41.933051 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:41.933019 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv"] Nov 03 17:17:41.939562 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:41.939540 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" Nov 03 17:17:41.960640 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:41.960616 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv"] Nov 03 17:17:41.965566 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:41.965120 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9c4762-serving-cert\"" Nov 03 17:17:42.065368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:42.065325 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1513cf28-6652-4788-a4eb-215a9cf87a77-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv\" (UID: \"1513cf28-6652-4788-a4eb-215a9cf87a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" Nov 03 17:17:42.065564 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:42.065455 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwspx\" (UniqueName: \"kubernetes.io/projected/1513cf28-6652-4788-a4eb-215a9cf87a77-kube-api-access-wwspx\") pod \"maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv\" (UID: \"1513cf28-6652-4788-a4eb-215a9cf87a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" Nov 03 17:17:42.065621 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:42.065609 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1513cf28-6652-4788-a4eb-215a9cf87a77-maas-api-tls\") pod \"maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv\" (UID: \"1513cf28-6652-4788-a4eb-215a9cf87a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" Nov 03 17:17:42.065789 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:42.065763 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1513cf28-6652-4788-a4eb-215a9cf87a77-metrics-certs\") pod \"maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv\" (UID: \"1513cf28-6652-4788-a4eb-215a9cf87a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" Nov 03 17:17:42.168845 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:42.168806 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1513cf28-6652-4788-a4eb-215a9cf87a77-maas-api-tls\") pod \"maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv\" (UID: \"1513cf28-6652-4788-a4eb-215a9cf87a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" Nov 03 17:17:42.174495 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:42.174452 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1513cf28-6652-4788-a4eb-215a9cf87a77-metrics-certs\") pod \"maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv\" (UID: \"1513cf28-6652-4788-a4eb-215a9cf87a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" Nov 03 17:17:42.175991 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:42.175964 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1513cf28-6652-4788-a4eb-215a9cf87a77-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv\" (UID: \"1513cf28-6652-4788-a4eb-215a9cf87a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" Nov 03 17:17:42.176952 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:42.176929 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwspx\" (UniqueName: \"kubernetes.io/projected/1513cf28-6652-4788-a4eb-215a9cf87a77-kube-api-access-wwspx\") pod \"maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv\" (UID: \"1513cf28-6652-4788-a4eb-215a9cf87a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" Nov 03 17:17:42.178793 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:42.178759 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1513cf28-6652-4788-a4eb-215a9cf87a77-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv\" (UID: \"1513cf28-6652-4788-a4eb-215a9cf87a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" Nov 03 17:17:42.179742 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:42.179720 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1513cf28-6652-4788-a4eb-215a9cf87a77-maas-api-tls\") pod \"maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv\" (UID: \"1513cf28-6652-4788-a4eb-215a9cf87a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" Nov 03 17:17:42.207629 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:42.207524 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1513cf28-6652-4788-a4eb-215a9cf87a77-metrics-certs\") pod \"maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv\" (UID: \"1513cf28-6652-4788-a4eb-215a9cf87a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" Nov 03 17:17:42.218107 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:42.218076 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwspx\" (UniqueName: \"kubernetes.io/projected/1513cf28-6652-4788-a4eb-215a9cf87a77-kube-api-access-wwspx\") pod \"maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv\" (UID: \"1513cf28-6652-4788-a4eb-215a9cf87a77\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" Nov 03 17:17:42.255962 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:42.255934 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" Nov 03 17:17:42.569737 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:17:42.569668 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1513cf28_6652_4788_a4eb_215a9cf87a77.slice/crio-1facd65e18bd691eeb59916d36dc0d1c8c1dbe8595f958523d1cdfac6951737f WatchSource:0}: Error finding container 1facd65e18bd691eeb59916d36dc0d1c8c1dbe8595f958523d1cdfac6951737f: Status 404 returned error can't find the container with id 1facd65e18bd691eeb59916d36dc0d1c8c1dbe8595f958523d1cdfac6951737f Nov 03 17:17:42.572117 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:42.572094 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv"] Nov 03 17:17:42.775216 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:42.775172 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" event={"ID":"1513cf28-6652-4788-a4eb-215a9cf87a77","Type":"ContainerStarted","Data":"1facd65e18bd691eeb59916d36dc0d1c8c1dbe8595f958523d1cdfac6951737f"} Nov 03 17:17:43.798313 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:43.798273 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" event={"ID":"1513cf28-6652-4788-a4eb-215a9cf87a77","Type":"ContainerStarted","Data":"3b978ff48dfb0627b9652e306545746182f1c0362f7c60da83b6e17465b63e68"} Nov 03 17:17:43.798808 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:43.798339 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" Nov 03 17:17:43.848309 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:43.848244 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" podStartSLOduration=2.489432385 podStartE2EDuration="2.848230518s" totalImagesPullingTime="358.798133ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:43.843506451 +0000 UTC m=+3807.432506419" watchObservedRunningTime="2026-11-03 17:17:43.848230518 +0000 UTC m=+3807.437230524" Nov 03 17:17:49.811035 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:49.811001 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" Nov 03 17:17:56.534070 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:56.534033 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg"] Nov 03 17:17:56.539363 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:56.539341 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" Nov 03 17:17:56.559134 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:56.559105 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-547925-serving-cert\"" Nov 03 17:17:56.572510 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:56.572479 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg"] Nov 03 17:17:56.727724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:56.727232 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b921c483-84fe-4ae5-8f29-f751b6c80ce5-metrics-certs\") pod \"maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg\" (UID: \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" Nov 03 17:17:56.727724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:56.727469 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b921c483-84fe-4ae5-8f29-f751b6c80ce5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg\" (UID: \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" Nov 03 17:17:56.727724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:56.727672 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b921c483-84fe-4ae5-8f29-f751b6c80ce5-maas-api-tls\") pod \"maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg\" (UID: \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" Nov 03 17:17:56.728121 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:56.727751 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjkrf\" (UniqueName: \"kubernetes.io/projected/b921c483-84fe-4ae5-8f29-f751b6c80ce5-kube-api-access-qjkrf\") pod \"maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg\" (UID: \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" Nov 03 17:17:56.829246 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:56.829152 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b921c483-84fe-4ae5-8f29-f751b6c80ce5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg\" (UID: \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" Nov 03 17:17:56.829449 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:56.829302 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b921c483-84fe-4ae5-8f29-f751b6c80ce5-maas-api-tls\") pod \"maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg\" (UID: \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" Nov 03 17:17:56.829449 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:56.829345 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjkrf\" (UniqueName: \"kubernetes.io/projected/b921c483-84fe-4ae5-8f29-f751b6c80ce5-kube-api-access-qjkrf\") pod \"maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg\" (UID: \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" Nov 03 17:17:56.829576 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:56.829479 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b921c483-84fe-4ae5-8f29-f751b6c80ce5-metrics-certs\") pod \"maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg\" (UID: \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" Nov 03 17:17:56.829916 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:56.829896 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b921c483-84fe-4ae5-8f29-f751b6c80ce5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg\" (UID: \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" Nov 03 17:17:56.832055 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:56.832030 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b921c483-84fe-4ae5-8f29-f751b6c80ce5-metrics-certs\") pod \"maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg\" (UID: \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" Nov 03 17:17:56.832180 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:56.832069 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b921c483-84fe-4ae5-8f29-f751b6c80ce5-maas-api-tls\") pod \"maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg\" (UID: \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" Nov 03 17:17:56.854644 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:56.854611 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjkrf\" (UniqueName: \"kubernetes.io/projected/b921c483-84fe-4ae5-8f29-f751b6c80ce5-kube-api-access-qjkrf\") pod \"maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg\" (UID: \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" Nov 03 17:17:56.858970 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:56.858943 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" Nov 03 17:17:57.172586 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:17:57.172548 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb921c483_84fe_4ae5_8f29_f751b6c80ce5.slice/crio-4711df3189750fff05b18dcd5a0f3f511df1b0cc64353d8c8177c0811a2b5ada WatchSource:0}: Error finding container 4711df3189750fff05b18dcd5a0f3f511df1b0cc64353d8c8177c0811a2b5ada: Status 404 returned error can't find the container with id 4711df3189750fff05b18dcd5a0f3f511df1b0cc64353d8c8177c0811a2b5ada Nov 03 17:17:57.175214 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:57.175192 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg"] Nov 03 17:17:57.957774 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:57.948801 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" event={"ID":"b921c483-84fe-4ae5-8f29-f751b6c80ce5","Type":"ContainerStarted","Data":"87292d465e496b99432d0f20141504a8ea4ceb6e314bb41a6db57300208c618f"} Nov 03 17:17:57.957774 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:57.948867 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" event={"ID":"b921c483-84fe-4ae5-8f29-f751b6c80ce5","Type":"ContainerStarted","Data":"4711df3189750fff05b18dcd5a0f3f511df1b0cc64353d8c8177c0811a2b5ada"} Nov 03 17:17:57.957774 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:57.949957 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" Nov 03 17:17:57.998058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:17:57.997993 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" podStartSLOduration=1.6630549719999999 podStartE2EDuration="1.997973021s" totalImagesPullingTime="334.918049ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:17:57.995845263 +0000 UTC m=+3821.584845230" watchObservedRunningTime="2026-11-03 17:17:57.997973021 +0000 UTC m=+3821.586972990" Nov 03 17:18:00.214863 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:18:00.214830 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:18:04.996363 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:18:04.996325 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" Nov 03 17:18:30.131016 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:18:30.130980 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:18:30.683733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:18:30.683632 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:19:00.143306 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:19:00.143259 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:19:13.986878 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:13.986840 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm"] Nov 03 17:19:13.993904 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:13.993873 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.015306 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.015277 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm"] Nov 03 17:19:14.019495 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.019457 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9c4762\"/\"openshift-service-ca.crt\"" Nov 03 17:19:14.020589 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.020555 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9c4762\"/\"kube-root-ca.crt\"" Nov 03 17:19:14.051333 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.051294 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9c4762\"/\"test-model-9c4762-kserve-self-signed-certs\"" Nov 03 17:19:14.051598 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.051575 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9c4762\"/\"default-dockercfg-h7rxr\"" Nov 03 17:19:14.094174 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.094145 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-tmp-dir\") pod \"test-model-9c4762-kserve-5787f6cbc7-qs6fm\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.094357 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.094341 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93035580-fe13-49e9-8c14-4878b6763fbd-tls-certs\") pod \"test-model-9c4762-kserve-5787f6cbc7-qs6fm\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.094513 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.094498 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-dshm\") pod \"test-model-9c4762-kserve-5787f6cbc7-qs6fm\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.094562 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.094544 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb2d6\" (UniqueName: \"kubernetes.io/projected/93035580-fe13-49e9-8c14-4878b6763fbd-kube-api-access-nb2d6\") pod \"test-model-9c4762-kserve-5787f6cbc7-qs6fm\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.094606 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.094587 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-model-cache\") pod \"test-model-9c4762-kserve-5787f6cbc7-qs6fm\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.094675 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.094656 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-home\") pod \"test-model-9c4762-kserve-5787f6cbc7-qs6fm\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.195961 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.195928 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93035580-fe13-49e9-8c14-4878b6763fbd-tls-certs\") pod \"test-model-9c4762-kserve-5787f6cbc7-qs6fm\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.196147 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.196080 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-dshm\") pod \"test-model-9c4762-kserve-5787f6cbc7-qs6fm\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.196147 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.196140 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb2d6\" (UniqueName: \"kubernetes.io/projected/93035580-fe13-49e9-8c14-4878b6763fbd-kube-api-access-nb2d6\") pod \"test-model-9c4762-kserve-5787f6cbc7-qs6fm\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.196276 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.196176 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-model-cache\") pod \"test-model-9c4762-kserve-5787f6cbc7-qs6fm\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.196276 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.196207 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-home\") pod \"test-model-9c4762-kserve-5787f6cbc7-qs6fm\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.196381 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.196338 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-tmp-dir\") pod \"test-model-9c4762-kserve-5787f6cbc7-qs6fm\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.196759 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.196730 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-home\") pod \"test-model-9c4762-kserve-5787f6cbc7-qs6fm\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.196917 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.196837 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-tmp-dir\") pod \"test-model-9c4762-kserve-5787f6cbc7-qs6fm\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.196917 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.196845 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-model-cache\") pod \"test-model-9c4762-kserve-5787f6cbc7-qs6fm\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.198446 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.198423 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-dshm\") pod \"test-model-9c4762-kserve-5787f6cbc7-qs6fm\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.198559 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.198495 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93035580-fe13-49e9-8c14-4878b6763fbd-tls-certs\") pod \"test-model-9c4762-kserve-5787f6cbc7-qs6fm\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.222245 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.222219 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb2d6\" (UniqueName: \"kubernetes.io/projected/93035580-fe13-49e9-8c14-4878b6763fbd-kube-api-access-nb2d6\") pod \"test-model-9c4762-kserve-5787f6cbc7-qs6fm\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.315096 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.315065 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:14.866774 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.866621 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm"] Nov 03 17:19:14.988651 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.988617 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" event={"ID":"93035580-fe13-49e9-8c14-4878b6763fbd","Type":"ContainerStarted","Data":"3ce3a7fcd9ee4f89711f78b47ec47a794ea96dec5475da4bae4c296ac49a9a97"} Nov 03 17:19:14.989102 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:14.988658 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" event={"ID":"93035580-fe13-49e9-8c14-4878b6763fbd","Type":"ContainerStarted","Data":"d662350294fa700e27163f7101a933a16ca9087164995d0a26eafb667e700d28"} Nov 03 17:19:15.038407 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:15.038353 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" podStartSLOduration=2.038341237 podStartE2EDuration="2.038341237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:19:15.036357423 +0000 UTC m=+3898.625357389" watchObservedRunningTime="2026-11-03 17:19:15.038341237 +0000 UTC m=+3898.627341233" Nov 03 17:19:15.315207 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:15.315175 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:17.202452 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:17.202427 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:19:17.205039 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:17.205017 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:19:21.633094 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.633060 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp"] Nov 03 17:19:21.638394 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.638372 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.660591 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.660561 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp"] Nov 03 17:19:21.664756 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.664722 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cb0fa3\"/\"default-dockercfg-22rqx\"" Nov 03 17:19:21.664917 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.664893 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cb0fa3\"/\"kube-root-ca.crt\"" Nov 03 17:19:21.665139 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.665121 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cb0fa3\"/\"test-model-cb0fa3-kserve-self-signed-certs\"" Nov 03 17:19:21.676446 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.676420 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cb0fa3\"/\"openshift-service-ca.crt\"" Nov 03 17:19:21.721769 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.721733 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xwtw\" (UniqueName: \"kubernetes.io/projected/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-kube-api-access-7xwtw\") pod \"test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.721926 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.721884 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-home\") pod \"test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.721971 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.721930 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-tls-certs\") pod \"test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.722050 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.722034 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-tmp-dir\") pod \"test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.722107 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.722095 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-dshm\") pod \"test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.722266 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.722253 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-model-cache\") pod \"test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.823739 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.823691 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-model-cache\") pod \"test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.823904 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.823841 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xwtw\" (UniqueName: \"kubernetes.io/projected/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-kube-api-access-7xwtw\") pod \"test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.823973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.823953 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-home\") pod \"test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.824037 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.823984 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-tls-certs\") pod \"test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.824037 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.824030 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-tmp-dir\") pod \"test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.824150 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.824047 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-model-cache\") pod \"test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.824150 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.824075 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-dshm\") pod \"test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.824384 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.824365 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-tmp-dir\") pod \"test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.824428 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.824367 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-home\") pod \"test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.826429 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.826410 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-dshm\") pod \"test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.826758 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.826741 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-tls-certs\") pod \"test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.848455 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.848431 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xwtw\" (UniqueName: \"kubernetes.io/projected/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-kube-api-access-7xwtw\") pod \"test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:21.964777 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:21.963935 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:22.429288 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:19:22.429254 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d4fef6b_5ca6_41c1_8809_a74e3c3401e1.slice/crio-9b571f290c3068768c2b69a5995292b51c20bc99ac19957816d7d2d2cba9d815 WatchSource:0}: Error finding container 9b571f290c3068768c2b69a5995292b51c20bc99ac19957816d7d2d2cba9d815: Status 404 returned error can't find the container with id 9b571f290c3068768c2b69a5995292b51c20bc99ac19957816d7d2d2cba9d815 Nov 03 17:19:22.430565 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:22.430540 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp"] Nov 03 17:19:22.628116 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:22.628085 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:19:23.137306 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:23.137262 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" event={"ID":"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1","Type":"ContainerStarted","Data":"eabbad8f9c5b027ead65747991cebb3bb2d6a910c6797b7ba601fc4c33935cee"} Nov 03 17:19:23.138657 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:23.137316 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" event={"ID":"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1","Type":"ContainerStarted","Data":"9b571f290c3068768c2b69a5995292b51c20bc99ac19957816d7d2d2cba9d815"} Nov 03 17:19:23.185136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:23.185085 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" podStartSLOduration=2.185070921 podStartE2EDuration="2.185070921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:19:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:19:23.183444628 +0000 UTC m=+3906.772444596" watchObservedRunningTime="2026-11-03 17:19:23.185070921 +0000 UTC m=+3906.774070888" Nov 03 17:19:23.969726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:23.966944 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:24.316682 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:24.315393 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:24.401719 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:24.401176 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:25.203724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:25.202859 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:19:30.131129 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:19:30.131054 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:19:31.965581 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:31.965549 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:31.982492 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:31.982465 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:32.225577 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:32.225488 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:19:34.310649 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.310613 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2"] Nov 03 17:19:34.318550 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.318518 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.332977 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.332922 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-model-cache\") pod \"test-model-547925-kserve-77c46fdc5-q9fz2\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.333401 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.332984 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-tmp-dir\") pod \"test-model-547925-kserve-77c46fdc5-q9fz2\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.333401 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.333148 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvvdx\" (UniqueName: \"kubernetes.io/projected/6318894b-7e77-47fa-805e-a28d53c1fead-kube-api-access-pvvdx\") pod \"test-model-547925-kserve-77c46fdc5-q9fz2\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.333401 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.333225 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6318894b-7e77-47fa-805e-a28d53c1fead-tls-certs\") pod \"test-model-547925-kserve-77c46fdc5-q9fz2\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.333401 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.333284 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-dshm\") pod \"test-model-547925-kserve-77c46fdc5-q9fz2\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.333401 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.333410 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-home\") pod \"test-model-547925-kserve-77c46fdc5-q9fz2\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.336461 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.336431 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2"] Nov 03 17:19:34.342310 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.342284 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-547925\"/\"default-dockercfg-smz7d\"" Nov 03 17:19:34.342407 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.342352 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-547925\"/\"openshift-service-ca.crt\"" Nov 03 17:19:34.349563 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.349541 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-547925\"/\"kube-root-ca.crt\"" Nov 03 17:19:34.349675 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.349565 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-547925\"/\"test-model-547925-kserve-self-signed-certs\"" Nov 03 17:19:34.435265 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.435210 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvvdx\" (UniqueName: \"kubernetes.io/projected/6318894b-7e77-47fa-805e-a28d53c1fead-kube-api-access-pvvdx\") pod \"test-model-547925-kserve-77c46fdc5-q9fz2\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.435481 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.435339 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6318894b-7e77-47fa-805e-a28d53c1fead-tls-certs\") pod \"test-model-547925-kserve-77c46fdc5-q9fz2\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.435481 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.435416 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-dshm\") pod \"test-model-547925-kserve-77c46fdc5-q9fz2\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.435606 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.435574 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-home\") pod \"test-model-547925-kserve-77c46fdc5-q9fz2\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.435761 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.435740 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-model-cache\") pod \"test-model-547925-kserve-77c46fdc5-q9fz2\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.435887 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.435788 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-tmp-dir\") pod \"test-model-547925-kserve-77c46fdc5-q9fz2\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.436048 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.436027 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-home\") pod \"test-model-547925-kserve-77c46fdc5-q9fz2\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.436121 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.436091 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-model-cache\") pod \"test-model-547925-kserve-77c46fdc5-q9fz2\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.436189 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.436163 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-tmp-dir\") pod \"test-model-547925-kserve-77c46fdc5-q9fz2\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.437913 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.437893 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-dshm\") pod \"test-model-547925-kserve-77c46fdc5-q9fz2\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.438479 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.438451 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6318894b-7e77-47fa-805e-a28d53c1fead-tls-certs\") pod \"test-model-547925-kserve-77c46fdc5-q9fz2\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.459996 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.459974 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvvdx\" (UniqueName: \"kubernetes.io/projected/6318894b-7e77-47fa-805e-a28d53c1fead-kube-api-access-pvvdx\") pod \"test-model-547925-kserve-77c46fdc5-q9fz2\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:34.642787 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:34.642451 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:35.054628 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:19:35.054589 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6318894b_7e77_47fa_805e_a28d53c1fead.slice/crio-d2ca02bbfddc8888af346474fff22f3e913696cf5cc15b66df13188b0f2b8ae4 WatchSource:0}: Error finding container d2ca02bbfddc8888af346474fff22f3e913696cf5cc15b66df13188b0f2b8ae4: Status 404 returned error can't find the container with id d2ca02bbfddc8888af346474fff22f3e913696cf5cc15b66df13188b0f2b8ae4 Nov 03 17:19:35.056219 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:35.056191 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2"] Nov 03 17:19:35.244457 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:35.244425 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" event={"ID":"6318894b-7e77-47fa-805e-a28d53c1fead","Type":"ContainerStarted","Data":"e36b4a77090bc5a35d346fb47bbe97104c501673790f36b78368080b06eb3c87"} Nov 03 17:19:35.244646 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:35.244466 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" event={"ID":"6318894b-7e77-47fa-805e-a28d53c1fead","Type":"ContainerStarted","Data":"d2ca02bbfddc8888af346474fff22f3e913696cf5cc15b66df13188b0f2b8ae4"} Nov 03 17:19:35.291690 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:35.291642 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" podStartSLOduration=1.291628585 podStartE2EDuration="1.291628585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:19:35.289001811 +0000 UTC m=+3918.878001778" watchObservedRunningTime="2026-11-03 17:19:35.291628585 +0000 UTC m=+3918.880628553" Nov 03 17:19:35.643607 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:35.643575 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:44.643922 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:44.643886 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:44.656673 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:44.656646 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:45.360541 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:45.360509 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:19:58.765253 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:58.765211 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj"] Nov 03 17:19:58.765854 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:19:58.765510 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" podUID="cf3d6e3e-c739-4f9f-982a-16856d80e08c" containerName="istio-proxy" containerID="cri-o://f402416917c0d17b1a7a2d8fe210291d129a2f007742bf708b4cc366288aee16" gracePeriod=30 Nov 03 17:20:00.132526 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:20:00.132486 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:20:03.461332 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:03.461293 4947 patch_prober.go:29] interesting pod/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:20:03.461907 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:03.461372 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" podUID="cf3d6e3e-c739-4f9f-982a-16856d80e08c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:20:04.075284 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.075254 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 17:20:04.094244 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.094210 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istiod-ca-cert\") pod \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " Nov 03 17:20:04.094471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.094318 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf3d6e3e-c739-4f9f-982a-16856d80e08c-kube-api-access-8vx6j\" (UniqueName: \"kubernetes.io/projected/cf3d6e3e-c739-4f9f-982a-16856d80e08c-kube-api-access-8vx6j\") pod \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " Nov 03 17:20:04.094645 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.094619 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-data\") pod \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " Nov 03 17:20:04.094839 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.094659 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istiod-ca-cert" pod "cf3d6e3e-c739-4f9f-982a-16856d80e08c" (UID: "cf3d6e3e-c739-4f9f-982a-16856d80e08c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:20:04.094839 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.094680 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-token\") pod \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " Nov 03 17:20:04.094839 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.094746 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-envoy\") pod \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " Nov 03 17:20:04.094839 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.094783 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-workload-socket\") pod \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " Nov 03 17:20:04.094839 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.094833 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-workload-certs\") pod \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " Nov 03 17:20:04.095201 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.094911 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-credential-socket\") pod \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " Nov 03 17:20:04.095201 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.094963 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-data" pod "cf3d6e3e-c739-4f9f-982a-16856d80e08c" (UID: "cf3d6e3e-c739-4f9f-982a-16856d80e08c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:04.095201 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.094975 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-podinfo\") pod \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\" (UID: \"cf3d6e3e-c739-4f9f-982a-16856d80e08c\") " Nov 03 17:20:04.095201 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.095029 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-workload-socket" pod "cf3d6e3e-c739-4f9f-982a-16856d80e08c" (UID: "cf3d6e3e-c739-4f9f-982a-16856d80e08c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:04.095392 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.095202 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-workload-certs" pod "cf3d6e3e-c739-4f9f-982a-16856d80e08c" (UID: "cf3d6e3e-c739-4f9f-982a-16856d80e08c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:04.095392 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.095217 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-credential-socket" pod "cf3d6e3e-c739-4f9f-982a-16856d80e08c" (UID: "cf3d6e3e-c739-4f9f-982a-16856d80e08c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:04.095864 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.095837 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:20:04.095864 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.095866 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:20:04.096036 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.095881 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:20:04.096036 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.095894 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:20:04.096036 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.095908 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:20:04.097213 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.097183 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-envoy" pod "cf3d6e3e-c739-4f9f-982a-16856d80e08c" (UID: "cf3d6e3e-c739-4f9f-982a-16856d80e08c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:04.097387 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.097360 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-podinfo" pod "cf3d6e3e-c739-4f9f-982a-16856d80e08c" (UID: "cf3d6e3e-c739-4f9f-982a-16856d80e08c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:20:04.097435 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.097393 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-token" pod "cf3d6e3e-c739-4f9f-982a-16856d80e08c" (UID: "cf3d6e3e-c739-4f9f-982a-16856d80e08c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:20:04.097435 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.097424 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3d6e3e-c739-4f9f-982a-16856d80e08c-kube-api-access-8vx6j" pod "cf3d6e3e-c739-4f9f-982a-16856d80e08c" (UID: "cf3d6e3e-c739-4f9f-982a-16856d80e08c"). InnerVolumeSpecName "kube-api-access-8vx6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:20:04.197414 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.197373 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vx6j\" (UniqueName: \"kubernetes.io/projected/cf3d6e3e-c739-4f9f-982a-16856d80e08c-kube-api-access-8vx6j\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:20:04.197414 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.197411 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:20:04.197414 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.197424 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:20:04.197678 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.197437 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf3d6e3e-c739-4f9f-982a-16856d80e08c-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:20:04.513383 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.513348 4947 generic.go:356] "Generic (PLEG): container finished" podID="cf3d6e3e-c739-4f9f-982a-16856d80e08c" containerID="f402416917c0d17b1a7a2d8fe210291d129a2f007742bf708b4cc366288aee16" exitCode=0 Nov 03 17:20:04.513974 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.513401 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" Nov 03 17:20:04.513974 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.513423 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" event={"ID":"cf3d6e3e-c739-4f9f-982a-16856d80e08c","Type":"ContainerDied","Data":"f402416917c0d17b1a7a2d8fe210291d129a2f007742bf708b4cc366288aee16"} Nov 03 17:20:04.513974 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.513475 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj" event={"ID":"cf3d6e3e-c739-4f9f-982a-16856d80e08c","Type":"ContainerDied","Data":"ee38db02d29b17ebad394cf10ce144be7b4308365218c324528db0e656bb42bb"} Nov 03 17:20:04.513974 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.513498 4947 scope.go:122] "RemoveContainer" containerID="f402416917c0d17b1a7a2d8fe210291d129a2f007742bf708b4cc366288aee16" Nov 03 17:20:04.526250 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.526227 4947 scope.go:122] "RemoveContainer" containerID="f402416917c0d17b1a7a2d8fe210291d129a2f007742bf708b4cc366288aee16" Nov 03 17:20:04.526572 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:20:04.526552 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f402416917c0d17b1a7a2d8fe210291d129a2f007742bf708b4cc366288aee16\": container with ID starting with f402416917c0d17b1a7a2d8fe210291d129a2f007742bf708b4cc366288aee16 not found: ID does not exist" containerID="f402416917c0d17b1a7a2d8fe210291d129a2f007742bf708b4cc366288aee16" Nov 03 17:20:04.526635 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.526583 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f402416917c0d17b1a7a2d8fe210291d129a2f007742bf708b4cc366288aee16"} err="failed to get container status \"f402416917c0d17b1a7a2d8fe210291d129a2f007742bf708b4cc366288aee16\": rpc error: code = NotFound desc = could not find container \"f402416917c0d17b1a7a2d8fe210291d129a2f007742bf708b4cc366288aee16\": container with ID starting with f402416917c0d17b1a7a2d8fe210291d129a2f007742bf708b4cc366288aee16 not found: ID does not exist" Nov 03 17:20:04.589764 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.589730 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj"] Nov 03 17:20:04.598268 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:04.598229 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-7t4kj"] Nov 03 17:20:06.568798 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:06.568764 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf3d6e3e-c739-4f9f-982a-16856d80e08c" path="/var/lib/kubelet/pods/cf3d6e3e-c739-4f9f-982a-16856d80e08c/volumes" Nov 03 17:20:11.836869 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:11.836835 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v"] Nov 03 17:20:11.837424 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:11.837046 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" podUID="732aa07c-3673-424e-a9a8-e7c2a5d5520a" containerName="istio-proxy" containerID="cri-o://88d1fe2bc1830f3f482b48194c727d28effabaaf6804cfd4a847ffb12deff4bd" gracePeriod=30 Nov 03 17:20:16.934198 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:16.934161 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Nov 03 17:20:16.934579 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:16.934232 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" podUID="732aa07c-3673-424e-a9a8-e7c2a5d5520a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Nov 03 17:20:17.140482 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.140447 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:20:17.182657 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.182621 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-credential-socket\") pod \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " Nov 03 17:20:17.182868 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.182780 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-data\") pod \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " Nov 03 17:20:17.182868 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.182836 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-workload-socket\") pod \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " Nov 03 17:20:17.183005 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.182987 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-token\" (UniqueName: \"kubernetes.io/projected/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-token\") pod \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " Nov 03 17:20:17.183062 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.182978 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-credential-socket" pod "732aa07c-3673-424e-a9a8-e7c2a5d5520a" (UID: "732aa07c-3673-424e-a9a8-e7c2a5d5520a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:17.183062 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.183055 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istiod-ca-cert\") pod \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " Nov 03 17:20:17.183143 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.183050 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-data" pod "732aa07c-3673-424e-a9a8-e7c2a5d5520a" (UID: "732aa07c-3673-424e-a9a8-e7c2a5d5520a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:17.183143 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.183074 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-workload-certs\") pod \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " Nov 03 17:20:17.183143 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.183122 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732aa07c-3673-424e-a9a8-e7c2a5d5520a-kube-api-access-pw59s\" (UniqueName: \"kubernetes.io/projected/732aa07c-3673-424e-a9a8-e7c2a5d5520a-kube-api-access-pw59s\") pod \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " Nov 03 17:20:17.183274 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.183148 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-workload-socket" pod "732aa07c-3673-424e-a9a8-e7c2a5d5520a" (UID: "732aa07c-3673-424e-a9a8-e7c2a5d5520a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:17.183274 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.183195 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-envoy\") pod \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " Nov 03 17:20:17.183274 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.183241 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-podinfo\") pod \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\" (UID: \"732aa07c-3673-424e-a9a8-e7c2a5d5520a\") " Nov 03 17:20:17.183453 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.183326 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-workload-certs" pod "732aa07c-3673-424e-a9a8-e7c2a5d5520a" (UID: "732aa07c-3673-424e-a9a8-e7c2a5d5520a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:17.183453 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.183362 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istiod-ca-cert" pod "732aa07c-3673-424e-a9a8-e7c2a5d5520a" (UID: "732aa07c-3673-424e-a9a8-e7c2a5d5520a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:20:17.184009 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.183985 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:20:17.184009 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.184010 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:20:17.184206 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.184023 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:20:17.184206 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.184034 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:20:17.184206 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.184046 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:20:17.185552 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.185523 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732aa07c-3673-424e-a9a8-e7c2a5d5520a-kube-api-access-pw59s" pod "732aa07c-3673-424e-a9a8-e7c2a5d5520a" (UID: "732aa07c-3673-424e-a9a8-e7c2a5d5520a"). InnerVolumeSpecName "kube-api-access-pw59s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:20:17.185665 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.185556 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-envoy" pod "732aa07c-3673-424e-a9a8-e7c2a5d5520a" (UID: "732aa07c-3673-424e-a9a8-e7c2a5d5520a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:20:17.185665 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.185610 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-token" pod "732aa07c-3673-424e-a9a8-e7c2a5d5520a" (UID: "732aa07c-3673-424e-a9a8-e7c2a5d5520a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:20:17.185907 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.185825 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-podinfo" pod "732aa07c-3673-424e-a9a8-e7c2a5d5520a" (UID: "732aa07c-3673-424e-a9a8-e7c2a5d5520a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:20:17.285973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.285937 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:20:17.285973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.285973 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw59s\" (UniqueName: \"kubernetes.io/projected/732aa07c-3673-424e-a9a8-e7c2a5d5520a-kube-api-access-pw59s\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:20:17.286185 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.285984 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:20:17.286185 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.285993 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/732aa07c-3673-424e-a9a8-e7c2a5d5520a-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:20:17.629800 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.629748 4947 generic.go:356] "Generic (PLEG): container finished" podID="732aa07c-3673-424e-a9a8-e7c2a5d5520a" containerID="88d1fe2bc1830f3f482b48194c727d28effabaaf6804cfd4a847ffb12deff4bd" exitCode=0 Nov 03 17:20:17.630019 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.629817 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" Nov 03 17:20:17.630019 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.629853 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" event={"ID":"732aa07c-3673-424e-a9a8-e7c2a5d5520a","Type":"ContainerDied","Data":"88d1fe2bc1830f3f482b48194c727d28effabaaf6804cfd4a847ffb12deff4bd"} Nov 03 17:20:17.630019 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.629917 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v" event={"ID":"732aa07c-3673-424e-a9a8-e7c2a5d5520a","Type":"ContainerDied","Data":"9d38aa8a2618dc109cb207acbc578f109ab6832ff89d39c49980cfca19850857"} Nov 03 17:20:17.630019 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.629945 4947 scope.go:122] "RemoveContainer" containerID="88d1fe2bc1830f3f482b48194c727d28effabaaf6804cfd4a847ffb12deff4bd" Nov 03 17:20:17.644268 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.644242 4947 scope.go:122] "RemoveContainer" containerID="88d1fe2bc1830f3f482b48194c727d28effabaaf6804cfd4a847ffb12deff4bd" Nov 03 17:20:17.644607 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:20:17.644587 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d1fe2bc1830f3f482b48194c727d28effabaaf6804cfd4a847ffb12deff4bd\": container with ID starting with 88d1fe2bc1830f3f482b48194c727d28effabaaf6804cfd4a847ffb12deff4bd not found: ID does not exist" containerID="88d1fe2bc1830f3f482b48194c727d28effabaaf6804cfd4a847ffb12deff4bd" Nov 03 17:20:17.644662 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.644615 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d1fe2bc1830f3f482b48194c727d28effabaaf6804cfd4a847ffb12deff4bd"} err="failed to get container status \"88d1fe2bc1830f3f482b48194c727d28effabaaf6804cfd4a847ffb12deff4bd\": rpc error: code = NotFound desc = could not find container \"88d1fe2bc1830f3f482b48194c727d28effabaaf6804cfd4a847ffb12deff4bd\": container with ID starting with 88d1fe2bc1830f3f482b48194c727d28effabaaf6804cfd4a847ffb12deff4bd not found: ID does not exist" Nov 03 17:20:17.690547 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.690509 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v"] Nov 03 17:20:17.703909 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:17.703868 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-xpk9v"] Nov 03 17:20:18.568660 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:18.568616 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="732aa07c-3673-424e-a9a8-e7c2a5d5520a" path="/var/lib/kubelet/pods/732aa07c-3673-424e-a9a8-e7c2a5d5520a/volumes" Nov 03 17:20:26.728137 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:20:26.728100 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:20:30.131391 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:20:30.131346 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:21:00.382113 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:21:00.382024 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:21:09.060026 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.059990 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7"] Nov 03 17:21:09.060970 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.060950 4947 cpu_manager.go:414] "containerMap: removing container" podUID="732aa07c-3673-424e-a9a8-e7c2a5d5520a" containerName="istio-proxy" Nov 03 17:21:09.060970 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.060970 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732aa07c-3673-424e-a9a8-e7c2a5d5520a" containerName="istio-proxy" Nov 03 17:21:09.061092 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.060992 4947 cpu_manager.go:414] "containerMap: removing container" podUID="cf3d6e3e-c739-4f9f-982a-16856d80e08c" containerName="istio-proxy" Nov 03 17:21:09.061092 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.060997 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf3d6e3e-c739-4f9f-982a-16856d80e08c" containerName="istio-proxy" Nov 03 17:21:09.061180 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.061170 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="732aa07c-3673-424e-a9a8-e7c2a5d5520a" containerName="istio-proxy" Nov 03 17:21:09.061221 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.061182 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf3d6e3e-c739-4f9f-982a-16856d80e08c" containerName="istio-proxy" Nov 03 17:21:09.065340 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.065321 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.104228 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.104196 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7"] Nov 03 17:21:09.157378 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.157345 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-envoy\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.157583 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.157403 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-data\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.157583 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.157505 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-workload-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.157583 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.157556 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istiod-ca-cert\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.157583 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.157580 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4svzn\" (UniqueName: \"kubernetes.io/projected/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-kube-api-access-4svzn\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.157840 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.157664 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-credential-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.157900 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.157884 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-workload-certs\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.157962 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.157946 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-podinfo\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.158069 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.158054 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-token\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.178634 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.178600 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-89df9a-openshift-default-dockercfg-hp8vk\"" Nov 03 17:21:09.259517 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.259485 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-envoy\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.259760 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.259545 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-data\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.259760 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.259577 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-workload-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.259760 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.259611 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istiod-ca-cert\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.259760 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.259634 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4svzn\" (UniqueName: \"kubernetes.io/projected/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-kube-api-access-4svzn\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.259760 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.259665 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-credential-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.260069 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.259818 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-workload-certs\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.260069 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.259880 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-podinfo\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.260069 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.260004 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-workload-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.260069 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.260025 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-token\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.260438 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.260076 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-data\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.260438 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.260185 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-credential-socket\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.260438 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.260249 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-workload-certs\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.260438 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.260383 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istiod-ca-cert\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.262029 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.262007 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-envoy\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.262556 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.262536 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-podinfo\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.284682 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.284647 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-token\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.289957 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.289932 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4svzn\" (UniqueName: \"kubernetes.io/projected/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-kube-api-access-4svzn\") pod \"e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.380268 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.380168 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:09.872357 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:21:09.872325 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc32f2f_8c40_45dd_b7cb_d93125ada2af.slice/crio-785c859272007136b5e70f49f13a1453a6929f4394a815964e1f6dbc8f4c8d64 WatchSource:0}: Error finding container 785c859272007136b5e70f49f13a1453a6929f4394a815964e1f6dbc8f4c8d64: Status 404 returned error can't find the container with id 785c859272007136b5e70f49f13a1453a6929f4394a815964e1f6dbc8f4c8d64 Nov 03 17:21:09.874053 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.874029 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7"] Nov 03 17:21:09.877086 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.877051 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:21:09.877178 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.877118 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:21:09.877178 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:09.877151 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:21:10.306313 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:10.306272 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" event={"ID":"0fc32f2f-8c40-45dd-b7cb-d93125ada2af","Type":"ContainerStarted","Data":"2d1394ba36af7770926b44553f1334dc89d27162f4139a16d8fbf2dba6d94597"} Nov 03 17:21:10.306816 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:10.306334 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" event={"ID":"0fc32f2f-8c40-45dd-b7cb-d93125ada2af","Type":"ContainerStarted","Data":"785c859272007136b5e70f49f13a1453a6929f4394a815964e1f6dbc8f4c8d64"} Nov 03 17:21:10.373192 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:10.373136 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" podStartSLOduration=1.373120414 podStartE2EDuration="1.373120414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:10.369558108 +0000 UTC m=+4013.958558075" watchObservedRunningTime="2026-11-03 17:21:10.373120414 +0000 UTC m=+4013.962120382" Nov 03 17:21:10.381255 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:10.381226 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:11.382200 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:11.382160 4947 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 17:21:11.382582 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:11.382210 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" podUID="0fc32f2f-8c40-45dd-b7cb-d93125ada2af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 17:21:12.380936 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:12.380907 4947 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 17:21:12.381131 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:12.380955 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" podUID="0fc32f2f-8c40-45dd-b7cb-d93125ada2af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 17:21:13.380955 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:13.380917 4947 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 17:21:13.381345 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:13.380983 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" podUID="0fc32f2f-8c40-45dd-b7cb-d93125ada2af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 17:21:14.380883 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:14.380849 4947 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 17:21:14.381067 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:14.380902 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" podUID="0fc32f2f-8c40-45dd-b7cb-d93125ada2af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 17:21:15.281477 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.281440 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc"] Nov 03 17:21:15.287365 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.287346 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.340150 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.340115 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc"] Nov 03 17:21:15.345789 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.344186 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-89df9a\"/\"default-dockercfg-tczqp\"" Nov 03 17:21:15.345789 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.344235 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-89df9a\"/\"kube-root-ca.crt\"" Nov 03 17:21:15.345789 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.344283 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-89df9a\"/\"openshift-service-ca.crt\"" Nov 03 17:21:15.345789 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.344329 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-89df9a\"/\"test-model-89df9a-kserve-self-signed-certs\"" Nov 03 17:21:15.380571 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.380535 4947 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 17:21:15.380826 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.380592 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" podUID="0fc32f2f-8c40-45dd-b7cb-d93125ada2af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 17:21:15.434592 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.434554 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-model-cache\") pod \"test-model-89df9a-kserve-85f7f879cc-cj9bc\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.435030 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.434754 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-tls-certs\") pod \"test-model-89df9a-kserve-85f7f879cc-cj9bc\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.435030 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.434856 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn8vv\" (UniqueName: \"kubernetes.io/projected/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-kube-api-access-xn8vv\") pod \"test-model-89df9a-kserve-85f7f879cc-cj9bc\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.435030 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.434915 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-home\") pod \"test-model-89df9a-kserve-85f7f879cc-cj9bc\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.435030 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.434967 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-dshm\") pod \"test-model-89df9a-kserve-85f7f879cc-cj9bc\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.435171 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.435049 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-tmp-dir\") pod \"test-model-89df9a-kserve-85f7f879cc-cj9bc\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.537063 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.536973 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-model-cache\") pod \"test-model-89df9a-kserve-85f7f879cc-cj9bc\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.537239 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.537108 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-tls-certs\") pod \"test-model-89df9a-kserve-85f7f879cc-cj9bc\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.537239 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.537184 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn8vv\" (UniqueName: \"kubernetes.io/projected/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-kube-api-access-xn8vv\") pod \"test-model-89df9a-kserve-85f7f879cc-cj9bc\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.537379 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.537248 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-home\") pod \"test-model-89df9a-kserve-85f7f879cc-cj9bc\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.537379 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.537306 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-dshm\") pod \"test-model-89df9a-kserve-85f7f879cc-cj9bc\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.537379 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.537327 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-tmp-dir\") pod \"test-model-89df9a-kserve-85f7f879cc-cj9bc\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.537530 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.537388 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-model-cache\") pod \"test-model-89df9a-kserve-85f7f879cc-cj9bc\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.537636 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.537609 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-home\") pod \"test-model-89df9a-kserve-85f7f879cc-cj9bc\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.537720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.537655 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-tmp-dir\") pod \"test-model-89df9a-kserve-85f7f879cc-cj9bc\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.539663 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.539640 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-dshm\") pod \"test-model-89df9a-kserve-85f7f879cc-cj9bc\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.539840 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.539825 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-tls-certs\") pod \"test-model-89df9a-kserve-85f7f879cc-cj9bc\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.595742 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.588884 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn8vv\" (UniqueName: \"kubernetes.io/projected/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-kube-api-access-xn8vv\") pod \"test-model-89df9a-kserve-85f7f879cc-cj9bc\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:15.638598 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:15.638327 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:16.132875 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:21:16.132844 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f6b355b_00f5_4ca1_8d73_4e5c7ca51cf9.slice/crio-7adaefba9c211bd603a869ce73af6607f3784c772b2c32ea67eb07ba65f05594 WatchSource:0}: Error finding container 7adaefba9c211bd603a869ce73af6607f3784c772b2c32ea67eb07ba65f05594: Status 404 returned error can't find the container with id 7adaefba9c211bd603a869ce73af6607f3784c772b2c32ea67eb07ba65f05594 Nov 03 17:21:16.134259 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:16.134239 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc"] Nov 03 17:21:16.380763 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:16.380618 4947 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body= Nov 03 17:21:16.380763 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:16.380676 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" podUID="0fc32f2f-8c40-45dd-b7cb-d93125ada2af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" Nov 03 17:21:16.381674 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:16.381644 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" event={"ID":"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9","Type":"ContainerStarted","Data":"546cdb81ea451bc8b3eefd4178b7f9d8132b958dc734d9cb308ed6b657ccd438"} Nov 03 17:21:16.381862 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:16.381691 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" event={"ID":"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9","Type":"ContainerStarted","Data":"7adaefba9c211bd603a869ce73af6607f3784c772b2c32ea67eb07ba65f05594"} Nov 03 17:21:16.431446 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:16.431387 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" podStartSLOduration=1.431371491 podStartE2EDuration="1.431371491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:21:16.426184825 +0000 UTC m=+4020.015184792" watchObservedRunningTime="2026-11-03 17:21:16.431371491 +0000 UTC m=+4020.020371457" Nov 03 17:21:16.640871 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:16.640786 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:17.386445 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:17.386406 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:17.386836 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:17.386818 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:17.387513 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:17.387487 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:21:25.640649 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:25.640619 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:25.653370 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:25.653345 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:26.476462 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:26.476434 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:21:30.132099 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:21:30.132058 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:21:45.227363 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:45.227329 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:21:53.715563 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:53.715529 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p"] Nov 03 17:21:53.717922 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:53.715802 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" podUID="54adffe4-51ee-41e0-8d1b-9d329330d512" containerName="istio-proxy" containerID="cri-o://42c6ce51a1845fda5be1f27c45c86adea0805919d8da4a41fb9dc7a4d5598067" gracePeriod=30 Nov 03 17:21:59.080707 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.080676 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:21:59.139328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.139293 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-podinfo\") pod \"54adffe4-51ee-41e0-8d1b-9d329330d512\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " Nov 03 17:21:59.139527 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.139377 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-token\" (UniqueName: \"kubernetes.io/projected/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-token\") pod \"54adffe4-51ee-41e0-8d1b-9d329330d512\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " Nov 03 17:21:59.139527 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.139515 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54adffe4-51ee-41e0-8d1b-9d329330d512-kube-api-access-rpl4b\" (UniqueName: \"kubernetes.io/projected/54adffe4-51ee-41e0-8d1b-9d329330d512-kube-api-access-rpl4b\") pod \"54adffe4-51ee-41e0-8d1b-9d329330d512\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " Nov 03 17:21:59.139643 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.139587 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-workload-certs\") pod \"54adffe4-51ee-41e0-8d1b-9d329330d512\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " Nov 03 17:21:59.139643 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.139624 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-data\") pod \"54adffe4-51ee-41e0-8d1b-9d329330d512\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " Nov 03 17:21:59.139779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.139715 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54adffe4-51ee-41e0-8d1b-9d329330d512-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54adffe4-51ee-41e0-8d1b-9d329330d512-istiod-ca-cert\") pod \"54adffe4-51ee-41e0-8d1b-9d329330d512\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " Nov 03 17:21:59.140015 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.139948 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-workload-certs" pod "54adffe4-51ee-41e0-8d1b-9d329330d512" (UID: "54adffe4-51ee-41e0-8d1b-9d329330d512"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:59.140015 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.139990 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-data" pod "54adffe4-51ee-41e0-8d1b-9d329330d512" (UID: "54adffe4-51ee-41e0-8d1b-9d329330d512"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:59.140197 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.140120 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54adffe4-51ee-41e0-8d1b-9d329330d512-istiod-ca-cert" pod "54adffe4-51ee-41e0-8d1b-9d329330d512" (UID: "54adffe4-51ee-41e0-8d1b-9d329330d512"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:21:59.140295 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.140279 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-credential-socket\") pod \"54adffe4-51ee-41e0-8d1b-9d329330d512\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " Nov 03 17:21:59.140378 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.140363 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-envoy\") pod \"54adffe4-51ee-41e0-8d1b-9d329330d512\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " Nov 03 17:21:59.140476 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.140458 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-workload-socket\") pod \"54adffe4-51ee-41e0-8d1b-9d329330d512\" (UID: \"54adffe4-51ee-41e0-8d1b-9d329330d512\") " Nov 03 17:21:59.140809 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.140686 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-credential-socket" pod "54adffe4-51ee-41e0-8d1b-9d329330d512" (UID: "54adffe4-51ee-41e0-8d1b-9d329330d512"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:59.140809 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.140791 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-workload-socket" pod "54adffe4-51ee-41e0-8d1b-9d329330d512" (UID: "54adffe4-51ee-41e0-8d1b-9d329330d512"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:59.141813 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.141789 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:21:59.142012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.141815 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:21:59.142012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.141829 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:21:59.142012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.141870 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54adffe4-51ee-41e0-8d1b-9d329330d512-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:21:59.142012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.141881 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:21:59.142430 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.142295 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-token" pod "54adffe4-51ee-41e0-8d1b-9d329330d512" (UID: "54adffe4-51ee-41e0-8d1b-9d329330d512"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:21:59.142759 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.142690 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-podinfo" pod "54adffe4-51ee-41e0-8d1b-9d329330d512" (UID: "54adffe4-51ee-41e0-8d1b-9d329330d512"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:21:59.142854 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.142807 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54adffe4-51ee-41e0-8d1b-9d329330d512-kube-api-access-rpl4b" pod "54adffe4-51ee-41e0-8d1b-9d329330d512" (UID: "54adffe4-51ee-41e0-8d1b-9d329330d512"). InnerVolumeSpecName "kube-api-access-rpl4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:21:59.143781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.143679 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-envoy" pod "54adffe4-51ee-41e0-8d1b-9d329330d512" (UID: "54adffe4-51ee-41e0-8d1b-9d329330d512"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:21:59.244253 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.244216 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:21:59.244253 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.244254 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:21:59.244535 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.244266 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54adffe4-51ee-41e0-8d1b-9d329330d512-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:21:59.244535 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.244274 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpl4b\" (UniqueName: \"kubernetes.io/projected/54adffe4-51ee-41e0-8d1b-9d329330d512-kube-api-access-rpl4b\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:21:59.755945 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.755911 4947 generic.go:356] "Generic (PLEG): container finished" podID="54adffe4-51ee-41e0-8d1b-9d329330d512" containerID="42c6ce51a1845fda5be1f27c45c86adea0805919d8da4a41fb9dc7a4d5598067" exitCode=0 Nov 03 17:21:59.756145 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.755979 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" Nov 03 17:21:59.756145 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.755989 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" event={"ID":"54adffe4-51ee-41e0-8d1b-9d329330d512","Type":"ContainerDied","Data":"42c6ce51a1845fda5be1f27c45c86adea0805919d8da4a41fb9dc7a4d5598067"} Nov 03 17:21:59.756145 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.756041 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p" event={"ID":"54adffe4-51ee-41e0-8d1b-9d329330d512","Type":"ContainerDied","Data":"9e14ca09b06db08c9797198e2ee98d790429050697d905ee52e1b584ed829bc4"} Nov 03 17:21:59.756145 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.756064 4947 scope.go:122] "RemoveContainer" containerID="42c6ce51a1845fda5be1f27c45c86adea0805919d8da4a41fb9dc7a4d5598067" Nov 03 17:21:59.768850 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.768832 4947 scope.go:122] "RemoveContainer" containerID="42c6ce51a1845fda5be1f27c45c86adea0805919d8da4a41fb9dc7a4d5598067" Nov 03 17:21:59.769073 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:21:59.769053 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c6ce51a1845fda5be1f27c45c86adea0805919d8da4a41fb9dc7a4d5598067\": container with ID starting with 42c6ce51a1845fda5be1f27c45c86adea0805919d8da4a41fb9dc7a4d5598067 not found: ID does not exist" containerID="42c6ce51a1845fda5be1f27c45c86adea0805919d8da4a41fb9dc7a4d5598067" Nov 03 17:21:59.769143 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.769084 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c6ce51a1845fda5be1f27c45c86adea0805919d8da4a41fb9dc7a4d5598067"} err="failed to get container status \"42c6ce51a1845fda5be1f27c45c86adea0805919d8da4a41fb9dc7a4d5598067\": rpc error: code = NotFound desc = could not find container \"42c6ce51a1845fda5be1f27c45c86adea0805919d8da4a41fb9dc7a4d5598067\": container with ID starting with 42c6ce51a1845fda5be1f27c45c86adea0805919d8da4a41fb9dc7a4d5598067 not found: ID does not exist" Nov 03 17:21:59.827970 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.827941 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p"] Nov 03 17:21:59.840928 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:21:59.840903 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e0e63d-openshift-default-5dc66f457-v8j5p"] Nov 03 17:22:00.131926 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:22:00.131845 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:22:00.607645 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:00.605099 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54adffe4-51ee-41e0-8d1b-9d329330d512" path="/var/lib/kubelet/pods/54adffe4-51ee-41e0-8d1b-9d329330d512/volumes" Nov 03 17:22:13.681277 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:13.681242 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn"] Nov 03 17:22:13.682240 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:13.682209 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" podUID="ea91c101-fb9f-4277-83d0-39229045f116" containerName="istio-proxy" containerID="cri-o://48abbff8f9e2c04e08f62c52581d8bc0a50f458a753f1a9a2a063116b5e4e710" gracePeriod=30 Nov 03 17:22:18.912761 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:18.912683 4947 generic.go:356] "Generic (PLEG): container finished" podID="ea91c101-fb9f-4277-83d0-39229045f116" containerID="48abbff8f9e2c04e08f62c52581d8bc0a50f458a753f1a9a2a063116b5e4e710" exitCode=0 Nov 03 17:22:18.913225 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:18.912905 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" event={"ID":"ea91c101-fb9f-4277-83d0-39229045f116","Type":"ContainerDied","Data":"48abbff8f9e2c04e08f62c52581d8bc0a50f458a753f1a9a2a063116b5e4e710"} Nov 03 17:22:19.067991 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.067963 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:22:19.114948 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.114909 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-workload-socket\") pod \"ea91c101-fb9f-4277-83d0-39229045f116\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " Nov 03 17:22:19.115150 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.115031 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea91c101-fb9f-4277-83d0-39229045f116-istio-token\" (UniqueName: \"kubernetes.io/projected/ea91c101-fb9f-4277-83d0-39229045f116-istio-token\") pod \"ea91c101-fb9f-4277-83d0-39229045f116\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " Nov 03 17:22:19.115150 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.115078 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-istio-data\") pod \"ea91c101-fb9f-4277-83d0-39229045f116\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " Nov 03 17:22:19.115274 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.115173 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-workload-socket" pod "ea91c101-fb9f-4277-83d0-39229045f116" (UID: "ea91c101-fb9f-4277-83d0-39229045f116"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:19.115274 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.115200 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea91c101-fb9f-4277-83d0-39229045f116-kube-api-access-4z7b8\" (UniqueName: \"kubernetes.io/projected/ea91c101-fb9f-4277-83d0-39229045f116-kube-api-access-4z7b8\") pod \"ea91c101-fb9f-4277-83d0-39229045f116\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " Nov 03 17:22:19.115382 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.115292 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea91c101-fb9f-4277-83d0-39229045f116-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea91c101-fb9f-4277-83d0-39229045f116-istio-podinfo\") pod \"ea91c101-fb9f-4277-83d0-39229045f116\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " Nov 03 17:22:19.115382 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.115346 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-credential-socket\") pod \"ea91c101-fb9f-4277-83d0-39229045f116\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " Nov 03 17:22:19.115490 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.115388 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea91c101-fb9f-4277-83d0-39229045f116-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea91c101-fb9f-4277-83d0-39229045f116-istiod-ca-cert\") pod \"ea91c101-fb9f-4277-83d0-39229045f116\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " Nov 03 17:22:19.115490 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.115458 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-istio-envoy\") pod \"ea91c101-fb9f-4277-83d0-39229045f116\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " Nov 03 17:22:19.115490 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.115385 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-istio-data" pod "ea91c101-fb9f-4277-83d0-39229045f116" (UID: "ea91c101-fb9f-4277-83d0-39229045f116"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:19.115650 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.115515 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-workload-certs\") pod \"ea91c101-fb9f-4277-83d0-39229045f116\" (UID: \"ea91c101-fb9f-4277-83d0-39229045f116\") " Nov 03 17:22:19.115650 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.115583 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-credential-socket" pod "ea91c101-fb9f-4277-83d0-39229045f116" (UID: "ea91c101-fb9f-4277-83d0-39229045f116"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:19.115784 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.115693 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea91c101-fb9f-4277-83d0-39229045f116-istiod-ca-cert" pod "ea91c101-fb9f-4277-83d0-39229045f116" (UID: "ea91c101-fb9f-4277-83d0-39229045f116"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:22:19.115931 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.115893 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-workload-certs" pod "ea91c101-fb9f-4277-83d0-39229045f116" (UID: "ea91c101-fb9f-4277-83d0-39229045f116"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:19.116519 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.116497 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:22:19.116581 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.116527 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:22:19.116581 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.116542 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:22:19.116581 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.116556 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea91c101-fb9f-4277-83d0-39229045f116-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:22:19.116581 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.116570 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:22:19.117763 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.117731 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-istio-envoy" pod "ea91c101-fb9f-4277-83d0-39229045f116" (UID: "ea91c101-fb9f-4277-83d0-39229045f116"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:19.117888 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.117807 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea91c101-fb9f-4277-83d0-39229045f116-istio-token" pod "ea91c101-fb9f-4277-83d0-39229045f116" (UID: "ea91c101-fb9f-4277-83d0-39229045f116"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:19.118017 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.117994 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea91c101-fb9f-4277-83d0-39229045f116-istio-podinfo" pod "ea91c101-fb9f-4277-83d0-39229045f116" (UID: "ea91c101-fb9f-4277-83d0-39229045f116"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:22:19.118078 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.118017 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea91c101-fb9f-4277-83d0-39229045f116-kube-api-access-4z7b8" pod "ea91c101-fb9f-4277-83d0-39229045f116" (UID: "ea91c101-fb9f-4277-83d0-39229045f116"). InnerVolumeSpecName "kube-api-access-4z7b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:19.218589 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.218555 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea91c101-fb9f-4277-83d0-39229045f116-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:22:19.218589 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.218586 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z7b8\" (UniqueName: \"kubernetes.io/projected/ea91c101-fb9f-4277-83d0-39229045f116-kube-api-access-4z7b8\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:22:19.218589 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.218597 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea91c101-fb9f-4277-83d0-39229045f116-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:22:19.218915 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.218605 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea91c101-fb9f-4277-83d0-39229045f116-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:22:19.922445 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.922418 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" Nov 03 17:22:19.922943 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.922451 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn" event={"ID":"ea91c101-fb9f-4277-83d0-39229045f116","Type":"ContainerDied","Data":"348d82c706e5ae246da8a5e0541e81502f385f8bb9bc60c6233111cac89355e6"} Nov 03 17:22:19.922943 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.922511 4947 scope.go:122] "RemoveContainer" containerID="48abbff8f9e2c04e08f62c52581d8bc0a50f458a753f1a9a2a063116b5e4e710" Nov 03 17:22:19.996424 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:19.996391 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn"] Nov 03 17:22:20.009443 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:20.009411 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d6fcae-openshift-default-6774bd965c-s65xn"] Nov 03 17:22:20.609670 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:20.605961 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea91c101-fb9f-4277-83d0-39229045f116" path="/var/lib/kubelet/pods/ea91c101-fb9f-4277-83d0-39229045f116/volumes" Nov 03 17:22:25.776388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:25.776349 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2"] Nov 03 17:22:25.776942 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:25.776582 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" podUID="e2b50359-b597-46ac-9ca1-af75415e1cf1" containerName="istio-proxy" containerID="cri-o://caa492307a1a50d1f625a08b675f0cc1e9884d6dc1b4ebf4114e40b128ab1bec" gracePeriod=30 Nov 03 17:22:29.451607 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:29.451523 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:22:29.451607 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:29.451591 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" podUID="e2b50359-b597-46ac-9ca1-af75415e1cf1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:22:30.131372 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:22:30.131331 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:22:31.034590 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.034552 4947 generic.go:356] "Generic (PLEG): container finished" podID="e2b50359-b597-46ac-9ca1-af75415e1cf1" containerID="caa492307a1a50d1f625a08b675f0cc1e9884d6dc1b4ebf4114e40b128ab1bec" exitCode=0 Nov 03 17:22:31.035052 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.034625 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" event={"ID":"e2b50359-b597-46ac-9ca1-af75415e1cf1","Type":"ContainerDied","Data":"caa492307a1a50d1f625a08b675f0cc1e9884d6dc1b4ebf4114e40b128ab1bec"} Nov 03 17:22:31.074628 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.074604 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:22:31.188681 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.188643 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-envoy\") pod \"e2b50359-b597-46ac-9ca1-af75415e1cf1\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " Nov 03 17:22:31.188868 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.188750 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-workload-certs\") pod \"e2b50359-b597-46ac-9ca1-af75415e1cf1\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " Nov 03 17:22:31.188924 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.188876 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-podinfo\") pod \"e2b50359-b597-46ac-9ca1-af75415e1cf1\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " Nov 03 17:22:31.188980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.188961 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2b50359-b597-46ac-9ca1-af75415e1cf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b50359-b597-46ac-9ca1-af75415e1cf1-istiod-ca-cert\") pod \"e2b50359-b597-46ac-9ca1-af75415e1cf1\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " Nov 03 17:22:31.189023 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.188997 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b50359-b597-46ac-9ca1-af75415e1cf1-kube-api-access-qpqwd\" (UniqueName: \"kubernetes.io/projected/e2b50359-b597-46ac-9ca1-af75415e1cf1-kube-api-access-qpqwd\") pod \"e2b50359-b597-46ac-9ca1-af75415e1cf1\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " Nov 03 17:22:31.189073 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.189042 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-workload-certs" pod "e2b50359-b597-46ac-9ca1-af75415e1cf1" (UID: "e2b50359-b597-46ac-9ca1-af75415e1cf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:31.189132 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.189087 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-workload-socket\") pod \"e2b50359-b597-46ac-9ca1-af75415e1cf1\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " Nov 03 17:22:31.189186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.189144 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-credential-socket\") pod \"e2b50359-b597-46ac-9ca1-af75415e1cf1\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " Nov 03 17:22:31.189186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.189165 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-data\") pod \"e2b50359-b597-46ac-9ca1-af75415e1cf1\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " Nov 03 17:22:31.189291 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.189189 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-token\" (UniqueName: \"kubernetes.io/projected/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-token\") pod \"e2b50359-b597-46ac-9ca1-af75415e1cf1\" (UID: \"e2b50359-b597-46ac-9ca1-af75415e1cf1\") " Nov 03 17:22:31.190193 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.189390 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-credential-socket" pod "e2b50359-b597-46ac-9ca1-af75415e1cf1" (UID: "e2b50359-b597-46ac-9ca1-af75415e1cf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:31.190193 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.189450 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b50359-b597-46ac-9ca1-af75415e1cf1-istiod-ca-cert" pod "e2b50359-b597-46ac-9ca1-af75415e1cf1" (UID: "e2b50359-b597-46ac-9ca1-af75415e1cf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:22:31.190193 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.189572 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-workload-socket" pod "e2b50359-b597-46ac-9ca1-af75415e1cf1" (UID: "e2b50359-b597-46ac-9ca1-af75415e1cf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:31.190193 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.189758 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-data" pod "e2b50359-b597-46ac-9ca1-af75415e1cf1" (UID: "e2b50359-b597-46ac-9ca1-af75415e1cf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:31.190193 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.190101 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:22:31.190193 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.190123 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:22:31.190193 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.190138 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:22:31.190193 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.190151 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b50359-b597-46ac-9ca1-af75415e1cf1-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:22:31.190193 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.190164 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:22:31.191587 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.191563 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-envoy" pod "e2b50359-b597-46ac-9ca1-af75415e1cf1" (UID: "e2b50359-b597-46ac-9ca1-af75415e1cf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:22:31.191789 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.191770 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-podinfo" pod "e2b50359-b597-46ac-9ca1-af75415e1cf1" (UID: "e2b50359-b597-46ac-9ca1-af75415e1cf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:22:31.191947 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.191927 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b50359-b597-46ac-9ca1-af75415e1cf1-kube-api-access-qpqwd" pod "e2b50359-b597-46ac-9ca1-af75415e1cf1" (UID: "e2b50359-b597-46ac-9ca1-af75415e1cf1"). InnerVolumeSpecName "kube-api-access-qpqwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:31.192306 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.192291 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-token" pod "e2b50359-b597-46ac-9ca1-af75415e1cf1" (UID: "e2b50359-b597-46ac-9ca1-af75415e1cf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:22:31.291674 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.291634 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpqwd\" (UniqueName: \"kubernetes.io/projected/e2b50359-b597-46ac-9ca1-af75415e1cf1-kube-api-access-qpqwd\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:22:31.291674 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.291669 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:22:31.291674 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.291680 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:22:31.291944 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:31.291691 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b50359-b597-46ac-9ca1-af75415e1cf1-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:22:32.064562 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:32.064531 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" Nov 03 17:22:32.064562 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:32.064554 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2" event={"ID":"e2b50359-b597-46ac-9ca1-af75415e1cf1","Type":"ContainerDied","Data":"ac1521be15b0b2f57ebb8fff60206e2b2f81a56461879954d495beaab363fdf8"} Nov 03 17:22:32.065123 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:32.064618 4947 scope.go:122] "RemoveContainer" containerID="caa492307a1a50d1f625a08b675f0cc1e9884d6dc1b4ebf4114e40b128ab1bec" Nov 03 17:22:32.140005 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:32.139974 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2"] Nov 03 17:22:32.153655 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:32.153622 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-ws7s2"] Nov 03 17:22:32.569476 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:22:32.569440 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2b50359-b597-46ac-9ca1-af75415e1cf1" path="/var/lib/kubelet/pods/e2b50359-b597-46ac-9ca1-af75415e1cf1/volumes" Nov 03 17:23:00.131563 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:23:00.131529 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:23:01.020921 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:23:01.020890 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:23:30.132225 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:23:30.132186 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:23:55.879469 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:23:55.879435 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm"] Nov 03 17:24:00.132130 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:24:00.132044 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:24:01.428759 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:01.428726 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6"] Nov 03 17:24:01.467126 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:01.467092 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv"] Nov 03 17:24:06.429062 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:06.429003 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" podUID="8c3d1351-007d-4fe5-be27-d868a8655c32" containerName="payload-processing" containerID="cri-o://3e8600e18ede2dff2a5d040b8efb4857acdced260c0001f074d83bf94dfbda59" gracePeriod=25 Nov 03 17:24:06.468068 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:06.468021 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" podUID="83746e7d-ec09-4142-b46b-38fa1471cc4a" containerName="payload-pre-processing" containerID="cri-o://d43a8352373cafbdc8f158fa1686efd34aad670cf38d224b6b89959f59c91c19" gracePeriod=25 Nov 03 17:24:10.284347 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:10.284307 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r"] Nov 03 17:24:10.552503 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:10.552414 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" podUID="83746e7d-ec09-4142-b46b-38fa1471cc4a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:24:10.553737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:10.553685 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" podUID="8c3d1351-007d-4fe5-be27-d868a8655c32" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:24:10.902584 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:10.902495 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" podUID="93035580-fe13-49e9-8c14-4878b6763fbd" containerName="main" containerID="cri-o://3ce3a7fcd9ee4f89711f78b47ec47a794ea96dec5475da4bae4c296ac49a9a97" gracePeriod=45 Nov 03 17:24:11.161145 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:11.158568 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" podUID="93035580-fe13-49e9-8c14-4878b6763fbd" containerName="main" probeResult="failure" output="Get \"https://10.132.1.8:8000/ready\": dial tcp 10.132.1.8:8000: connect: connection refused" Nov 03 17:24:11.772485 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:11.772458 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:24:11.905581 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:11.905499 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93035580-fe13-49e9-8c14-4878b6763fbd-kube-api-access-nb2d6\" (UniqueName: \"kubernetes.io/projected/93035580-fe13-49e9-8c14-4878b6763fbd-kube-api-access-nb2d6\") pod \"93035580-fe13-49e9-8c14-4878b6763fbd\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " Nov 03 17:24:11.905791 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:11.905594 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-model-cache\") pod \"93035580-fe13-49e9-8c14-4878b6763fbd\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " Nov 03 17:24:11.905791 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:11.905722 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-tmp-dir\") pod \"93035580-fe13-49e9-8c14-4878b6763fbd\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " Nov 03 17:24:11.905791 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:11.905775 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93035580-fe13-49e9-8c14-4878b6763fbd-tls-certs\" (UniqueName: \"kubernetes.io/secret/93035580-fe13-49e9-8c14-4878b6763fbd-tls-certs\") pod \"93035580-fe13-49e9-8c14-4878b6763fbd\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " Nov 03 17:24:11.905984 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:11.905820 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-home\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-home\") pod \"93035580-fe13-49e9-8c14-4878b6763fbd\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " Nov 03 17:24:11.905984 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:11.905850 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-dshm\") pod \"93035580-fe13-49e9-8c14-4878b6763fbd\" (UID: \"93035580-fe13-49e9-8c14-4878b6763fbd\") " Nov 03 17:24:11.905984 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:11.905921 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-model-cache" pod "93035580-fe13-49e9-8c14-4878b6763fbd" (UID: "93035580-fe13-49e9-8c14-4878b6763fbd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:11.906125 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:11.906105 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-home" pod "93035580-fe13-49e9-8c14-4878b6763fbd" (UID: "93035580-fe13-49e9-8c14-4878b6763fbd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:11.906272 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:11.906245 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-tmp-dir" pod "93035580-fe13-49e9-8c14-4878b6763fbd" (UID: "93035580-fe13-49e9-8c14-4878b6763fbd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:11.906566 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:11.906545 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:11.906631 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:11.906570 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:11.906631 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:11.906578 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:11.908005 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:11.907983 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93035580-fe13-49e9-8c14-4878b6763fbd-kube-api-access-nb2d6" pod "93035580-fe13-49e9-8c14-4878b6763fbd" (UID: "93035580-fe13-49e9-8c14-4878b6763fbd"). InnerVolumeSpecName "kube-api-access-nb2d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:11.908494 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:11.908476 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93035580-fe13-49e9-8c14-4878b6763fbd-tls-certs" pod "93035580-fe13-49e9-8c14-4878b6763fbd" (UID: "93035580-fe13-49e9-8c14-4878b6763fbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:24:11.908569 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:11.908489 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-dshm" pod "93035580-fe13-49e9-8c14-4878b6763fbd" (UID: "93035580-fe13-49e9-8c14-4878b6763fbd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:12.008385 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:12.008347 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb2d6\" (UniqueName: \"kubernetes.io/projected/93035580-fe13-49e9-8c14-4878b6763fbd-kube-api-access-nb2d6\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:12.008385 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:12.008379 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93035580-fe13-49e9-8c14-4878b6763fbd-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:12.008385 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:12.008388 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93035580-fe13-49e9-8c14-4878b6763fbd-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:12.031517 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:12.031485 4947 generic.go:356] "Generic (PLEG): container finished" podID="93035580-fe13-49e9-8c14-4878b6763fbd" containerID="3ce3a7fcd9ee4f89711f78b47ec47a794ea96dec5475da4bae4c296ac49a9a97" exitCode=0 Nov 03 17:24:12.031691 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:12.031530 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" Nov 03 17:24:12.031691 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:12.031572 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" event={"ID":"93035580-fe13-49e9-8c14-4878b6763fbd","Type":"ContainerDied","Data":"3ce3a7fcd9ee4f89711f78b47ec47a794ea96dec5475da4bae4c296ac49a9a97"} Nov 03 17:24:12.031691 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:12.031616 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm" event={"ID":"93035580-fe13-49e9-8c14-4878b6763fbd","Type":"ContainerDied","Data":"d662350294fa700e27163f7101a933a16ca9087164995d0a26eafb667e700d28"} Nov 03 17:24:12.031691 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:12.031640 4947 scope.go:122] "RemoveContainer" containerID="3ce3a7fcd9ee4f89711f78b47ec47a794ea96dec5475da4bae4c296ac49a9a97" Nov 03 17:24:12.044439 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:12.044417 4947 scope.go:122] "RemoveContainer" containerID="3ce3a7fcd9ee4f89711f78b47ec47a794ea96dec5475da4bae4c296ac49a9a97" Nov 03 17:24:12.044776 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:24:12.044756 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce3a7fcd9ee4f89711f78b47ec47a794ea96dec5475da4bae4c296ac49a9a97\": container with ID starting with 3ce3a7fcd9ee4f89711f78b47ec47a794ea96dec5475da4bae4c296ac49a9a97 not found: ID does not exist" containerID="3ce3a7fcd9ee4f89711f78b47ec47a794ea96dec5475da4bae4c296ac49a9a97" Nov 03 17:24:12.044849 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:12.044783 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce3a7fcd9ee4f89711f78b47ec47a794ea96dec5475da4bae4c296ac49a9a97"} err="failed to get container status \"3ce3a7fcd9ee4f89711f78b47ec47a794ea96dec5475da4bae4c296ac49a9a97\": rpc error: code = NotFound desc = could not find container \"3ce3a7fcd9ee4f89711f78b47ec47a794ea96dec5475da4bae4c296ac49a9a97\": container with ID starting with 3ce3a7fcd9ee4f89711f78b47ec47a794ea96dec5475da4bae4c296ac49a9a97 not found: ID does not exist" Nov 03 17:24:12.089843 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:12.089811 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm"] Nov 03 17:24:12.103112 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:12.103087 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9c4762/test-model-9c4762-kserve-5787f6cbc7-qs6fm"] Nov 03 17:24:12.567934 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:12.567903 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93035580-fe13-49e9-8c14-4878b6763fbd" path="/var/lib/kubelet/pods/93035580-fe13-49e9-8c14-4878b6763fbd/volumes" Nov 03 17:24:15.284754 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:15.284690 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" podUID="458486ff-035b-44c6-853c-c6b071e5370f" containerName="payload-pre-processing" containerID="cri-o://accac995b88726ac8f9715dfd000ebab0b3314a88bd2db9d39e9884288f78bfb" gracePeriod=25 Nov 03 17:24:17.256849 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:17.256821 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:24:17.269235 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:17.269206 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:24:18.416983 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:18.416943 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:24:20.546552 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:20.546514 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" podUID="83746e7d-ec09-4142-b46b-38fa1471cc4a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:24:20.551587 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:20.551545 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" podUID="8c3d1351-007d-4fe5-be27-d868a8655c32" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:24:23.326374 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:23.326335 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" podUID="458486ff-035b-44c6-853c-c6b071e5370f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:24:26.752867 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:26.752829 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t"] Nov 03 17:24:26.753650 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:26.753112 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" podUID="769d68a5-7504-4c6a-9db1-355bff4c7ef8" containerName="maas-api" containerID="cri-o://b6894f3f36d8cd3264ad55436210533c36e3a035e68bc0b0f136d04a11e3f418" gracePeriod=30 Nov 03 17:24:26.849449 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:26.849422 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" Nov 03 17:24:26.869138 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:26.869114 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" Nov 03 17:24:26.900555 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:26.900526 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c3d1351-007d-4fe5-be27-d868a8655c32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c3d1351-007d-4fe5-be27-d868a8655c32-plugins-config-volume\") pod \"8c3d1351-007d-4fe5-be27-d868a8655c32\" (UID: \"8c3d1351-007d-4fe5-be27-d868a8655c32\") " Nov 03 17:24:26.900804 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:26.900661 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c3d1351-007d-4fe5-be27-d868a8655c32-kube-api-access-5nkdc\" (UniqueName: \"kubernetes.io/projected/8c3d1351-007d-4fe5-be27-d868a8655c32-kube-api-access-5nkdc\") pod \"8c3d1351-007d-4fe5-be27-d868a8655c32\" (UID: \"8c3d1351-007d-4fe5-be27-d868a8655c32\") " Nov 03 17:24:26.901016 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:26.900987 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3d1351-007d-4fe5-be27-d868a8655c32-plugins-config-volume" pod "8c3d1351-007d-4fe5-be27-d868a8655c32" (UID: "8c3d1351-007d-4fe5-be27-d868a8655c32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:24:26.901616 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:26.901587 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c3d1351-007d-4fe5-be27-d868a8655c32-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:26.903442 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:26.903396 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3d1351-007d-4fe5-be27-d868a8655c32-kube-api-access-5nkdc" pod "8c3d1351-007d-4fe5-be27-d868a8655c32" (UID: "8c3d1351-007d-4fe5-be27-d868a8655c32"). InnerVolumeSpecName "kube-api-access-5nkdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:27.028726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.027672 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83746e7d-ec09-4142-b46b-38fa1471cc4a-kube-api-access-x47ms\" (UniqueName: \"kubernetes.io/projected/83746e7d-ec09-4142-b46b-38fa1471cc4a-kube-api-access-x47ms\") pod \"83746e7d-ec09-4142-b46b-38fa1471cc4a\" (UID: \"83746e7d-ec09-4142-b46b-38fa1471cc4a\") " Nov 03 17:24:27.028726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.027855 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83746e7d-ec09-4142-b46b-38fa1471cc4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83746e7d-ec09-4142-b46b-38fa1471cc4a-plugins-config-volume\") pod \"83746e7d-ec09-4142-b46b-38fa1471cc4a\" (UID: \"83746e7d-ec09-4142-b46b-38fa1471cc4a\") " Nov 03 17:24:27.095715 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.077595 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83746e7d-ec09-4142-b46b-38fa1471cc4a-kube-api-access-x47ms" pod "83746e7d-ec09-4142-b46b-38fa1471cc4a" (UID: "83746e7d-ec09-4142-b46b-38fa1471cc4a"). InnerVolumeSpecName "kube-api-access-x47ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:27.124188 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.123894 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nkdc\" (UniqueName: \"kubernetes.io/projected/8c3d1351-007d-4fe5-be27-d868a8655c32-kube-api-access-5nkdc\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:27.124188 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.123924 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x47ms\" (UniqueName: \"kubernetes.io/projected/83746e7d-ec09-4142-b46b-38fa1471cc4a-kube-api-access-x47ms\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:27.124447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.124209 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83746e7d-ec09-4142-b46b-38fa1471cc4a-plugins-config-volume" pod "83746e7d-ec09-4142-b46b-38fa1471cc4a" (UID: "83746e7d-ec09-4142-b46b-38fa1471cc4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:24:27.286769 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.253242 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83746e7d-ec09-4142-b46b-38fa1471cc4a-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:27.445782 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.444492 4947 generic.go:356] "Generic (PLEG): container finished" podID="83746e7d-ec09-4142-b46b-38fa1471cc4a" containerID="d43a8352373cafbdc8f158fa1686efd34aad670cf38d224b6b89959f59c91c19" exitCode=0 Nov 03 17:24:27.445782 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.444546 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" Nov 03 17:24:27.445782 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.444638 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" event={"ID":"83746e7d-ec09-4142-b46b-38fa1471cc4a","Type":"ContainerDied","Data":"d43a8352373cafbdc8f158fa1686efd34aad670cf38d224b6b89959f59c91c19"} Nov 03 17:24:27.445782 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.445557 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv" event={"ID":"83746e7d-ec09-4142-b46b-38fa1471cc4a","Type":"ContainerDied","Data":"2557f44df31e7220457498fc55fcfdac0923b21e0bfdf1f5daf2ab0979d2c82c"} Nov 03 17:24:27.445782 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.445609 4947 scope.go:122] "RemoveContainer" containerID="d43a8352373cafbdc8f158fa1686efd34aad670cf38d224b6b89959f59c91c19" Nov 03 17:24:27.451958 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.448752 4947 generic.go:356] "Generic (PLEG): container finished" podID="8c3d1351-007d-4fe5-be27-d868a8655c32" containerID="3e8600e18ede2dff2a5d040b8efb4857acdced260c0001f074d83bf94dfbda59" exitCode=0 Nov 03 17:24:27.451958 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.448976 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" event={"ID":"8c3d1351-007d-4fe5-be27-d868a8655c32","Type":"ContainerDied","Data":"3e8600e18ede2dff2a5d040b8efb4857acdced260c0001f074d83bf94dfbda59"} Nov 03 17:24:27.451958 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.449020 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" event={"ID":"8c3d1351-007d-4fe5-be27-d868a8655c32","Type":"ContainerDied","Data":"4e399d74862b94d36240238be54b0f366c4210435b2afce14b16ddcdcdd2b1d5"} Nov 03 17:24:27.451958 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.449096 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6" Nov 03 17:24:27.464821 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.463714 4947 generic.go:356] "Generic (PLEG): container finished" podID="769d68a5-7504-4c6a-9db1-355bff4c7ef8" containerID="b6894f3f36d8cd3264ad55436210533c36e3a035e68bc0b0f136d04a11e3f418" exitCode=0 Nov 03 17:24:27.464821 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.463809 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" event={"ID":"769d68a5-7504-4c6a-9db1-355bff4c7ef8","Type":"ContainerDied","Data":"b6894f3f36d8cd3264ad55436210533c36e3a035e68bc0b0f136d04a11e3f418"} Nov 03 17:24:27.478889 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.478867 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" Nov 03 17:24:27.480791 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.480774 4947 scope.go:122] "RemoveContainer" containerID="d43a8352373cafbdc8f158fa1686efd34aad670cf38d224b6b89959f59c91c19" Nov 03 17:24:27.481069 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:24:27.481049 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d43a8352373cafbdc8f158fa1686efd34aad670cf38d224b6b89959f59c91c19\": container with ID starting with d43a8352373cafbdc8f158fa1686efd34aad670cf38d224b6b89959f59c91c19 not found: ID does not exist" containerID="d43a8352373cafbdc8f158fa1686efd34aad670cf38d224b6b89959f59c91c19" Nov 03 17:24:27.481131 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.481077 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d43a8352373cafbdc8f158fa1686efd34aad670cf38d224b6b89959f59c91c19"} err="failed to get container status \"d43a8352373cafbdc8f158fa1686efd34aad670cf38d224b6b89959f59c91c19\": rpc error: code = NotFound desc = could not find container \"d43a8352373cafbdc8f158fa1686efd34aad670cf38d224b6b89959f59c91c19\": container with ID starting with d43a8352373cafbdc8f158fa1686efd34aad670cf38d224b6b89959f59c91c19 not found: ID does not exist" Nov 03 17:24:27.481131 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.481093 4947 scope.go:122] "RemoveContainer" containerID="3e8600e18ede2dff2a5d040b8efb4857acdced260c0001f074d83bf94dfbda59" Nov 03 17:24:27.491273 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.491245 4947 scope.go:122] "RemoveContainer" containerID="3e8600e18ede2dff2a5d040b8efb4857acdced260c0001f074d83bf94dfbda59" Nov 03 17:24:27.491527 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:24:27.491512 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e8600e18ede2dff2a5d040b8efb4857acdced260c0001f074d83bf94dfbda59\": container with ID starting with 3e8600e18ede2dff2a5d040b8efb4857acdced260c0001f074d83bf94dfbda59 not found: ID does not exist" containerID="3e8600e18ede2dff2a5d040b8efb4857acdced260c0001f074d83bf94dfbda59" Nov 03 17:24:27.491568 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.491537 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e8600e18ede2dff2a5d040b8efb4857acdced260c0001f074d83bf94dfbda59"} err="failed to get container status \"3e8600e18ede2dff2a5d040b8efb4857acdced260c0001f074d83bf94dfbda59\": rpc error: code = NotFound desc = could not find container \"3e8600e18ede2dff2a5d040b8efb4857acdced260c0001f074d83bf94dfbda59\": container with ID starting with 3e8600e18ede2dff2a5d040b8efb4857acdced260c0001f074d83bf94dfbda59 not found: ID does not exist" Nov 03 17:24:27.523075 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.523040 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6"] Nov 03 17:24:27.525021 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.525003 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/769d68a5-7504-4c6a-9db1-355bff4c7ef8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/769d68a5-7504-4c6a-9db1-355bff4c7ef8-maas-api-tls\") pod \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\" (UID: \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\") " Nov 03 17:24:27.525112 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.525100 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/769d68a5-7504-4c6a-9db1-355bff4c7ef8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/769d68a5-7504-4c6a-9db1-355bff4c7ef8-metrics-certs\") pod \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\" (UID: \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\") " Nov 03 17:24:27.525152 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.525125 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769d68a5-7504-4c6a-9db1-355bff4c7ef8-kube-api-access-hv5wr\" (UniqueName: \"kubernetes.io/projected/769d68a5-7504-4c6a-9db1-355bff4c7ef8-kube-api-access-hv5wr\") pod \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\" (UID: \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\") " Nov 03 17:24:27.525313 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.525298 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/769d68a5-7504-4c6a-9db1-355bff4c7ef8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/769d68a5-7504-4c6a-9db1-355bff4c7ef8-openshift-service-ca\") pod \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\" (UID: \"769d68a5-7504-4c6a-9db1-355bff4c7ef8\") " Nov 03 17:24:27.525679 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.525610 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769d68a5-7504-4c6a-9db1-355bff4c7ef8-openshift-service-ca" pod "769d68a5-7504-4c6a-9db1-355bff4c7ef8" (UID: "769d68a5-7504-4c6a-9db1-355bff4c7ef8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:24:27.526185 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.526162 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/769d68a5-7504-4c6a-9db1-355bff4c7ef8-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:27.527272 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.527250 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/769d68a5-7504-4c6a-9db1-355bff4c7ef8-metrics-certs" pod "769d68a5-7504-4c6a-9db1-355bff4c7ef8" (UID: "769d68a5-7504-4c6a-9db1-355bff4c7ef8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:24:27.527372 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.527331 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/769d68a5-7504-4c6a-9db1-355bff4c7ef8-maas-api-tls" pod "769d68a5-7504-4c6a-9db1-355bff4c7ef8" (UID: "769d68a5-7504-4c6a-9db1-355bff4c7ef8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:24:27.527485 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.527468 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769d68a5-7504-4c6a-9db1-355bff4c7ef8-kube-api-access-hv5wr" pod "769d68a5-7504-4c6a-9db1-355bff4c7ef8" (UID: "769d68a5-7504-4c6a-9db1-355bff4c7ef8"). InnerVolumeSpecName "kube-api-access-hv5wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:27.536305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.536282 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-fc40cc-7df4c6d9cf-6sgl6"] Nov 03 17:24:27.611972 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.611940 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv"] Nov 03 17:24:27.619012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.618986 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-fc40cc-9c5496699-bdbqv"] Nov 03 17:24:27.628491 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.628460 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/769d68a5-7504-4c6a-9db1-355bff4c7ef8-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:27.628491 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.628486 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/769d68a5-7504-4c6a-9db1-355bff4c7ef8-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:27.628636 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:27.628496 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv5wr\" (UniqueName: \"kubernetes.io/projected/769d68a5-7504-4c6a-9db1-355bff4c7ef8-kube-api-access-hv5wr\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:28.475539 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:28.475508 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" Nov 03 17:24:28.475539 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:28.475523 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t" event={"ID":"769d68a5-7504-4c6a-9db1-355bff4c7ef8","Type":"ContainerDied","Data":"548818cc2d44d7c699aeedf49ee93d32bce463beb7a17473328cf05c7ae3f08a"} Nov 03 17:24:28.476168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:28.475596 4947 scope.go:122] "RemoveContainer" containerID="b6894f3f36d8cd3264ad55436210533c36e3a035e68bc0b0f136d04a11e3f418" Nov 03 17:24:28.539433 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:28.539404 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t"] Nov 03 17:24:28.551683 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:28.551658 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-fc40cc-6d75847d7-7k96t"] Nov 03 17:24:28.568328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:28.568298 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="769d68a5-7504-4c6a-9db1-355bff4c7ef8" path="/var/lib/kubelet/pods/769d68a5-7504-4c6a-9db1-355bff4c7ef8/volumes" Nov 03 17:24:28.568821 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:28.568803 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83746e7d-ec09-4142-b46b-38fa1471cc4a" path="/var/lib/kubelet/pods/83746e7d-ec09-4142-b46b-38fa1471cc4a/volumes" Nov 03 17:24:28.569251 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:28.569236 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c3d1351-007d-4fe5-be27-d868a8655c32" path="/var/lib/kubelet/pods/8c3d1351-007d-4fe5-be27-d868a8655c32/volumes" Nov 03 17:24:30.132980 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:24:30.132939 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:24:33.326796 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:33.326760 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" podUID="458486ff-035b-44c6-853c-c6b071e5370f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:24:33.390521 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:33.390491 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682"] Nov 03 17:24:33.390754 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:33.390730 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" podUID="629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" containerName="istio-proxy" containerID="cri-o://4292063619ab9038f48bf110d15c1f918027deaff9231717a14404efd2692760" gracePeriod=30 Nov 03 17:24:35.541235 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:35.541149 4947 generic.go:356] "Generic (PLEG): container finished" podID="458486ff-035b-44c6-853c-c6b071e5370f" containerID="accac995b88726ac8f9715dfd000ebab0b3314a88bd2db9d39e9884288f78bfb" exitCode=0 Nov 03 17:24:35.541687 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:35.541341 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" event={"ID":"458486ff-035b-44c6-853c-c6b071e5370f","Type":"ContainerDied","Data":"accac995b88726ac8f9715dfd000ebab0b3314a88bd2db9d39e9884288f78bfb"} Nov 03 17:24:35.608386 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:35.608349 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" Nov 03 17:24:35.694230 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:35.694076 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/458486ff-035b-44c6-853c-c6b071e5370f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/458486ff-035b-44c6-853c-c6b071e5370f-plugins-config-volume\") pod \"458486ff-035b-44c6-853c-c6b071e5370f\" (UID: \"458486ff-035b-44c6-853c-c6b071e5370f\") " Nov 03 17:24:35.695937 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:35.694524 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/458486ff-035b-44c6-853c-c6b071e5370f-kube-api-access-xqkkq\" (UniqueName: \"kubernetes.io/projected/458486ff-035b-44c6-853c-c6b071e5370f-kube-api-access-xqkkq\") pod \"458486ff-035b-44c6-853c-c6b071e5370f\" (UID: \"458486ff-035b-44c6-853c-c6b071e5370f\") " Nov 03 17:24:35.698369 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:35.698086 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/458486ff-035b-44c6-853c-c6b071e5370f-plugins-config-volume" pod "458486ff-035b-44c6-853c-c6b071e5370f" (UID: "458486ff-035b-44c6-853c-c6b071e5370f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:24:35.730733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:35.730195 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55"] Nov 03 17:24:35.730733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:35.730313 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/458486ff-035b-44c6-853c-c6b071e5370f-kube-api-access-xqkkq" pod "458486ff-035b-44c6-853c-c6b071e5370f" (UID: "458486ff-035b-44c6-853c-c6b071e5370f"). InnerVolumeSpecName "kube-api-access-xqkkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:35.730733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:35.730426 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" podUID="1508d21f-1858-4232-be82-bcda731862b9" containerName="istio-proxy" containerID="cri-o://0e75f3ace41c88506258a0b364ffc210f286edb0f4a52affa3d6112078255f04" gracePeriod=30 Nov 03 17:24:35.906850 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:35.905081 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/458486ff-035b-44c6-853c-c6b071e5370f-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:35.906850 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:35.905116 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqkkq\" (UniqueName: \"kubernetes.io/projected/458486ff-035b-44c6-853c-c6b071e5370f-kube-api-access-xqkkq\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:36.553298 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:36.553266 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" Nov 03 17:24:36.553820 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:36.553330 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r" event={"ID":"458486ff-035b-44c6-853c-c6b071e5370f","Type":"ContainerDied","Data":"223c1551c287249bac6463a152e85fed42bd2d681cac9c547c16144d6eddfaa0"} Nov 03 17:24:36.553820 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:36.553422 4947 scope.go:122] "RemoveContainer" containerID="accac995b88726ac8f9715dfd000ebab0b3314a88bd2db9d39e9884288f78bfb" Nov 03 17:24:36.656527 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:36.656497 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r"] Nov 03 17:24:36.665844 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:36.665807 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-24cb6d-757cb9587f-rld8r"] Nov 03 17:24:38.588120 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.588083 4947 generic.go:356] "Generic (PLEG): container finished" podID="629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" containerID="4292063619ab9038f48bf110d15c1f918027deaff9231717a14404efd2692760" exitCode=0 Nov 03 17:24:38.593652 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.593626 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="458486ff-035b-44c6-853c-c6b071e5370f" path="/var/lib/kubelet/pods/458486ff-035b-44c6-853c-c6b071e5370f/volumes" Nov 03 17:24:38.594031 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.594012 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" event={"ID":"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009","Type":"ContainerDied","Data":"4292063619ab9038f48bf110d15c1f918027deaff9231717a14404efd2692760"} Nov 03 17:24:38.732431 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.732397 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:24:38.772294 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.772261 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-token\" (UniqueName: \"kubernetes.io/projected/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-token\") pod \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " Nov 03 17:24:38.772455 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.772382 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-kube-api-access-xfmkn\" (UniqueName: \"kubernetes.io/projected/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-kube-api-access-xfmkn\") pod \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " Nov 03 17:24:38.772455 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.772422 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-data\") pod \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " Nov 03 17:24:38.772537 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.772476 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-envoy\") pod \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " Nov 03 17:24:38.772572 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.772565 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-podinfo\") pod \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " Nov 03 17:24:38.772635 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.772616 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-credential-socket\") pod \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " Nov 03 17:24:38.772809 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.772727 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-workload-certs\") pod \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " Nov 03 17:24:38.772809 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.772769 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-workload-socket\") pod \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " Nov 03 17:24:38.772809 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.772782 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-data" pod "629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" (UID: "629a8dc0-af1f-4e22-aeaa-e2fa86a4a009"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:38.773482 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.773300 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-credential-socket" pod "629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" (UID: "629a8dc0-af1f-4e22-aeaa-e2fa86a4a009"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:38.773482 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.773444 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-workload-certs" pod "629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" (UID: "629a8dc0-af1f-4e22-aeaa-e2fa86a4a009"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:38.774499 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.774475 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-workload-socket" pod "629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" (UID: "629a8dc0-af1f-4e22-aeaa-e2fa86a4a009"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:38.775054 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.774503 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:38.775054 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.774936 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:38.775054 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.774964 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:38.775054 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.774968 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-token" pod "629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" (UID: "629a8dc0-af1f-4e22-aeaa-e2fa86a4a009"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:38.775054 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.775037 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-kube-api-access-xfmkn" pod "629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" (UID: "629a8dc0-af1f-4e22-aeaa-e2fa86a4a009"). InnerVolumeSpecName "kube-api-access-xfmkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:38.775310 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.775132 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-podinfo" pod "629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" (UID: "629a8dc0-af1f-4e22-aeaa-e2fa86a4a009"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:24:38.775310 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.775300 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-envoy" pod "629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" (UID: "629a8dc0-af1f-4e22-aeaa-e2fa86a4a009"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:38.876599 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.876566 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istiod-ca-cert\") pod \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\" (UID: \"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009\") " Nov 03 17:24:38.877012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.876984 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istiod-ca-cert" pod "629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" (UID: "629a8dc0-af1f-4e22-aeaa-e2fa86a4a009"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:24:38.877509 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.877490 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:38.877547 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.877513 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:38.877547 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.877521 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfmkn\" (UniqueName: \"kubernetes.io/projected/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-kube-api-access-xfmkn\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:38.877547 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.877533 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:38.877547 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.877542 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:38.877680 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:38.877549 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:39.165853 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:39.165819 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6"] Nov 03 17:24:39.166042 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:39.166023 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" podUID="e7c02310-bff9-4d86-b641-8022d891b395" containerName="istio-proxy" containerID="cri-o://25ecc4cc8c564e3de6e8805bd72df40a63354305526c3413dcad51df711ac170" gracePeriod=30 Nov 03 17:24:39.597954 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:39.597927 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" Nov 03 17:24:39.598395 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:39.597922 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682" event={"ID":"629a8dc0-af1f-4e22-aeaa-e2fa86a4a009","Type":"ContainerDied","Data":"be40fca6fe680d9fcb8665c406a5223421c73b7e1a5d2ca8f46568efa2b35ac0"} Nov 03 17:24:39.598395 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:39.598069 4947 scope.go:122] "RemoveContainer" containerID="4292063619ab9038f48bf110d15c1f918027deaff9231717a14404efd2692760" Nov 03 17:24:39.658839 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:39.658809 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682"] Nov 03 17:24:39.673326 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:39.673303 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-l2682"] Nov 03 17:24:40.568927 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:40.568898 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" path="/var/lib/kubelet/pods/629a8dc0-af1f-4e22-aeaa-e2fa86a4a009/volumes" Nov 03 17:24:41.039624 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.039600 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:24:41.115580 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.115386 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-istio-data\") pod \"1508d21f-1858-4232-be82-bcda731862b9\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " Nov 03 17:24:41.115580 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.115541 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1508d21f-1858-4232-be82-bcda731862b9-istio-token\" (UniqueName: \"kubernetes.io/projected/1508d21f-1858-4232-be82-bcda731862b9-istio-token\") pod \"1508d21f-1858-4232-be82-bcda731862b9\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " Nov 03 17:24:41.115866 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.115658 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-istio-envoy\") pod \"1508d21f-1858-4232-be82-bcda731862b9\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " Nov 03 17:24:41.115866 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.115748 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-workload-certs\") pod \"1508d21f-1858-4232-be82-bcda731862b9\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " Nov 03 17:24:41.115866 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.115744 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-istio-data" pod "1508d21f-1858-4232-be82-bcda731862b9" (UID: "1508d21f-1858-4232-be82-bcda731862b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:41.115866 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.115832 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1508d21f-1858-4232-be82-bcda731862b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1508d21f-1858-4232-be82-bcda731862b9-istiod-ca-cert\") pod \"1508d21f-1858-4232-be82-bcda731862b9\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " Nov 03 17:24:41.116079 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.115880 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1508d21f-1858-4232-be82-bcda731862b9-kube-api-access-rrdkl\" (UniqueName: \"kubernetes.io/projected/1508d21f-1858-4232-be82-bcda731862b9-kube-api-access-rrdkl\") pod \"1508d21f-1858-4232-be82-bcda731862b9\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " Nov 03 17:24:41.116079 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.115958 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-credential-socket\") pod \"1508d21f-1858-4232-be82-bcda731862b9\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " Nov 03 17:24:41.116079 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.116020 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1508d21f-1858-4232-be82-bcda731862b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1508d21f-1858-4232-be82-bcda731862b9-istio-podinfo\") pod \"1508d21f-1858-4232-be82-bcda731862b9\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " Nov 03 17:24:41.116234 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.116082 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-workload-socket\") pod \"1508d21f-1858-4232-be82-bcda731862b9\" (UID: \"1508d21f-1858-4232-be82-bcda731862b9\") " Nov 03 17:24:41.116234 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.116117 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-workload-certs" pod "1508d21f-1858-4232-be82-bcda731862b9" (UID: "1508d21f-1858-4232-be82-bcda731862b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:41.116234 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.116172 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1508d21f-1858-4232-be82-bcda731862b9-istiod-ca-cert" pod "1508d21f-1858-4232-be82-bcda731862b9" (UID: "1508d21f-1858-4232-be82-bcda731862b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:24:41.116391 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.116359 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-credential-socket" pod "1508d21f-1858-4232-be82-bcda731862b9" (UID: "1508d21f-1858-4232-be82-bcda731862b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:41.117000 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.116937 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:41.117000 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.116938 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-workload-socket" pod "1508d21f-1858-4232-be82-bcda731862b9" (UID: "1508d21f-1858-4232-be82-bcda731862b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:41.117000 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.116961 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1508d21f-1858-4232-be82-bcda731862b9-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:41.117000 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.116974 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:41.117000 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.116991 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:41.119114 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.119089 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1508d21f-1858-4232-be82-bcda731862b9-istio-podinfo" pod "1508d21f-1858-4232-be82-bcda731862b9" (UID: "1508d21f-1858-4232-be82-bcda731862b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:24:41.121180 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.121148 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1508d21f-1858-4232-be82-bcda731862b9-kube-api-access-rrdkl" pod "1508d21f-1858-4232-be82-bcda731862b9" (UID: "1508d21f-1858-4232-be82-bcda731862b9"). InnerVolumeSpecName "kube-api-access-rrdkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:41.121180 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.121167 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-istio-envoy" pod "1508d21f-1858-4232-be82-bcda731862b9" (UID: "1508d21f-1858-4232-be82-bcda731862b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:41.121523 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.121505 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1508d21f-1858-4232-be82-bcda731862b9-istio-token" pod "1508d21f-1858-4232-be82-bcda731862b9" (UID: "1508d21f-1858-4232-be82-bcda731862b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:41.218951 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.218920 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrdkl\" (UniqueName: \"kubernetes.io/projected/1508d21f-1858-4232-be82-bcda731862b9-kube-api-access-rrdkl\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:41.218951 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.218954 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1508d21f-1858-4232-be82-bcda731862b9-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:41.219183 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.218965 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:41.219183 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.218972 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1508d21f-1858-4232-be82-bcda731862b9-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:41.219183 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.218980 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1508d21f-1858-4232-be82-bcda731862b9-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:41.616775 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.616732 4947 generic.go:356] "Generic (PLEG): container finished" podID="1508d21f-1858-4232-be82-bcda731862b9" containerID="0e75f3ace41c88506258a0b364ffc210f286edb0f4a52affa3d6112078255f04" exitCode=0 Nov 03 17:24:41.616963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.616794 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" Nov 03 17:24:41.616963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.616806 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" event={"ID":"1508d21f-1858-4232-be82-bcda731862b9","Type":"ContainerDied","Data":"0e75f3ace41c88506258a0b364ffc210f286edb0f4a52affa3d6112078255f04"} Nov 03 17:24:41.616963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.616863 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55" event={"ID":"1508d21f-1858-4232-be82-bcda731862b9","Type":"ContainerDied","Data":"cc13355cb6537493a68c5c76ae2349df40dc351856247edc2288c050568e7c6f"} Nov 03 17:24:41.616963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.616894 4947 scope.go:122] "RemoveContainer" containerID="0e75f3ace41c88506258a0b364ffc210f286edb0f4a52affa3d6112078255f04" Nov 03 17:24:41.630252 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.630232 4947 scope.go:122] "RemoveContainer" containerID="03f58fdf817c9e0ca7390c901b4a4953c998c940c379b158ad2c2a1ea80e9245" Nov 03 17:24:41.640950 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.640922 4947 scope.go:122] "RemoveContainer" containerID="0e75f3ace41c88506258a0b364ffc210f286edb0f4a52affa3d6112078255f04" Nov 03 17:24:41.641263 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:24:41.641233 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e75f3ace41c88506258a0b364ffc210f286edb0f4a52affa3d6112078255f04\": container with ID starting with 0e75f3ace41c88506258a0b364ffc210f286edb0f4a52affa3d6112078255f04 not found: ID does not exist" containerID="0e75f3ace41c88506258a0b364ffc210f286edb0f4a52affa3d6112078255f04" Nov 03 17:24:41.641348 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.641275 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e75f3ace41c88506258a0b364ffc210f286edb0f4a52affa3d6112078255f04"} err="failed to get container status \"0e75f3ace41c88506258a0b364ffc210f286edb0f4a52affa3d6112078255f04\": rpc error: code = NotFound desc = could not find container \"0e75f3ace41c88506258a0b364ffc210f286edb0f4a52affa3d6112078255f04\": container with ID starting with 0e75f3ace41c88506258a0b364ffc210f286edb0f4a52affa3d6112078255f04 not found: ID does not exist" Nov 03 17:24:41.641348 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.641299 4947 scope.go:122] "RemoveContainer" containerID="03f58fdf817c9e0ca7390c901b4a4953c998c940c379b158ad2c2a1ea80e9245" Nov 03 17:24:41.641579 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:24:41.641556 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f58fdf817c9e0ca7390c901b4a4953c998c940c379b158ad2c2a1ea80e9245\": container with ID starting with 03f58fdf817c9e0ca7390c901b4a4953c998c940c379b158ad2c2a1ea80e9245 not found: ID does not exist" containerID="03f58fdf817c9e0ca7390c901b4a4953c998c940c379b158ad2c2a1ea80e9245" Nov 03 17:24:41.641622 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.641588 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f58fdf817c9e0ca7390c901b4a4953c998c940c379b158ad2c2a1ea80e9245"} err="failed to get container status \"03f58fdf817c9e0ca7390c901b4a4953c998c940c379b158ad2c2a1ea80e9245\": rpc error: code = NotFound desc = could not find container \"03f58fdf817c9e0ca7390c901b4a4953c998c940c379b158ad2c2a1ea80e9245\": container with ID starting with 03f58fdf817c9e0ca7390c901b4a4953c998c940c379b158ad2c2a1ea80e9245 not found: ID does not exist" Nov 03 17:24:41.689038 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.688999 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55"] Nov 03 17:24:41.702366 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:41.702332 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fc40cc-openshift-default-598c5986f7-6db55"] Nov 03 17:24:42.569856 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:42.569820 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1508d21f-1858-4232-be82-bcda731862b9" path="/var/lib/kubelet/pods/1508d21f-1858-4232-be82-bcda731862b9/volumes" Nov 03 17:24:42.637091 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:42.637061 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr"] Nov 03 17:24:42.637337 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:42.637274 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" podUID="d9f0e126-f94a-4cb6-b187-e05f23476696" containerName="istio-proxy" containerID="cri-o://3dff21d654da0f2332a9f2530d080428f0be198405c87d22cfb457c427e2b694" gracePeriod=30 Nov 03 17:24:43.859976 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:43.859940 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt"] Nov 03 17:24:44.459191 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.459166 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:24:44.589843 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.579150 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c02310-bff9-4d86-b641-8022d891b395-kube-api-access-cjrr9\" (UniqueName: \"kubernetes.io/projected/e7c02310-bff9-4d86-b641-8022d891b395-kube-api-access-cjrr9\") pod \"e7c02310-bff9-4d86-b641-8022d891b395\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " Nov 03 17:24:44.589843 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.579237 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7c02310-bff9-4d86-b641-8022d891b395-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c02310-bff9-4d86-b641-8022d891b395-istio-podinfo\") pod \"e7c02310-bff9-4d86-b641-8022d891b395\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " Nov 03 17:24:44.589843 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.579302 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-credential-socket\") pod \"e7c02310-bff9-4d86-b641-8022d891b395\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " Nov 03 17:24:44.589843 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.579353 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-istio-data\") pod \"e7c02310-bff9-4d86-b641-8022d891b395\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " Nov 03 17:24:44.589843 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.579389 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7c02310-bff9-4d86-b641-8022d891b395-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c02310-bff9-4d86-b641-8022d891b395-istiod-ca-cert\") pod \"e7c02310-bff9-4d86-b641-8022d891b395\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " Nov 03 17:24:44.589843 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.579511 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-workload-socket\") pod \"e7c02310-bff9-4d86-b641-8022d891b395\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " Nov 03 17:24:44.589843 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.579598 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7c02310-bff9-4d86-b641-8022d891b395-istio-token\" (UniqueName: \"kubernetes.io/projected/e7c02310-bff9-4d86-b641-8022d891b395-istio-token\") pod \"e7c02310-bff9-4d86-b641-8022d891b395\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " Nov 03 17:24:44.589843 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.579639 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-workload-certs\") pod \"e7c02310-bff9-4d86-b641-8022d891b395\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " Nov 03 17:24:44.614724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.591176 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-istio-data" pod "e7c02310-bff9-4d86-b641-8022d891b395" (UID: "e7c02310-bff9-4d86-b641-8022d891b395"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:44.614724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.591762 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-workload-socket" pod "e7c02310-bff9-4d86-b641-8022d891b395" (UID: "e7c02310-bff9-4d86-b641-8022d891b395"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:44.614724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.592024 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7c02310-bff9-4d86-b641-8022d891b395-istiod-ca-cert" pod "e7c02310-bff9-4d86-b641-8022d891b395" (UID: "e7c02310-bff9-4d86-b641-8022d891b395"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:24:44.614724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.594532 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-workload-certs" pod "e7c02310-bff9-4d86-b641-8022d891b395" (UID: "e7c02310-bff9-4d86-b641-8022d891b395"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:44.614724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.594757 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-credential-socket" pod "e7c02310-bff9-4d86-b641-8022d891b395" (UID: "e7c02310-bff9-4d86-b641-8022d891b395"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:44.614724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.594802 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-istio-envoy\") pod \"e7c02310-bff9-4d86-b641-8022d891b395\" (UID: \"e7c02310-bff9-4d86-b641-8022d891b395\") " Nov 03 17:24:44.614724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.595785 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:44.614724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.595809 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:44.614724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.595823 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:44.614724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.595836 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:44.614724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.595848 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7c02310-bff9-4d86-b641-8022d891b395-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:44.640959 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.637032 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7c02310-bff9-4d86-b641-8022d891b395-istio-podinfo" pod "e7c02310-bff9-4d86-b641-8022d891b395" (UID: "e7c02310-bff9-4d86-b641-8022d891b395"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:24:44.640959 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.637223 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c02310-bff9-4d86-b641-8022d891b395-istio-token" pod "e7c02310-bff9-4d86-b641-8022d891b395" (UID: "e7c02310-bff9-4d86-b641-8022d891b395"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:44.640959 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.637336 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7c02310-bff9-4d86-b641-8022d891b395-kube-api-access-cjrr9" pod "e7c02310-bff9-4d86-b641-8022d891b395" (UID: "e7c02310-bff9-4d86-b641-8022d891b395"). InnerVolumeSpecName "kube-api-access-cjrr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:44.675129 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.675090 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-istio-envoy" pod "e7c02310-bff9-4d86-b641-8022d891b395" (UID: "e7c02310-bff9-4d86-b641-8022d891b395"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:44.699772 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.689869 4947 generic.go:356] "Generic (PLEG): container finished" podID="e7c02310-bff9-4d86-b641-8022d891b395" containerID="25ecc4cc8c564e3de6e8805bd72df40a63354305526c3413dcad51df711ac170" exitCode=0 Nov 03 17:24:44.699772 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.690038 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" event={"ID":"e7c02310-bff9-4d86-b641-8022d891b395","Type":"ContainerDied","Data":"25ecc4cc8c564e3de6e8805bd72df40a63354305526c3413dcad51df711ac170"} Nov 03 17:24:44.699772 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.690087 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" event={"ID":"e7c02310-bff9-4d86-b641-8022d891b395","Type":"ContainerDied","Data":"449dedc0ec25986c870defdf07c4124f76d72711164ed82b3baaabed146e1f94"} Nov 03 17:24:44.699772 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.690115 4947 scope.go:122] "RemoveContainer" containerID="25ecc4cc8c564e3de6e8805bd72df40a63354305526c3413dcad51df711ac170" Nov 03 17:24:44.699772 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.690268 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6" Nov 03 17:24:44.718508 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.718481 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7c02310-bff9-4d86-b641-8022d891b395-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:44.718689 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.718674 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7c02310-bff9-4d86-b641-8022d891b395-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:44.718844 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.718828 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjrr9\" (UniqueName: \"kubernetes.io/projected/e7c02310-bff9-4d86-b641-8022d891b395-kube-api-access-cjrr9\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:44.718967 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.718952 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7c02310-bff9-4d86-b641-8022d891b395-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:44.738935 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.738909 4947 scope.go:122] "RemoveContainer" containerID="25ecc4cc8c564e3de6e8805bd72df40a63354305526c3413dcad51df711ac170" Nov 03 17:24:44.739274 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:24:44.739254 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25ecc4cc8c564e3de6e8805bd72df40a63354305526c3413dcad51df711ac170\": container with ID starting with 25ecc4cc8c564e3de6e8805bd72df40a63354305526c3413dcad51df711ac170 not found: ID does not exist" containerID="25ecc4cc8c564e3de6e8805bd72df40a63354305526c3413dcad51df711ac170" Nov 03 17:24:44.739454 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.739287 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25ecc4cc8c564e3de6e8805bd72df40a63354305526c3413dcad51df711ac170"} err="failed to get container status \"25ecc4cc8c564e3de6e8805bd72df40a63354305526c3413dcad51df711ac170\": rpc error: code = NotFound desc = could not find container \"25ecc4cc8c564e3de6e8805bd72df40a63354305526c3413dcad51df711ac170\": container with ID starting with 25ecc4cc8c564e3de6e8805bd72df40a63354305526c3413dcad51df711ac170 not found: ID does not exist" Nov 03 17:24:44.786201 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.786166 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6"] Nov 03 17:24:44.793960 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.793931 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-3a0e27-openshift-default-6bc6f466fc-bwdv6"] Nov 03 17:24:44.917652 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.917598 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst"] Nov 03 17:24:44.918289 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:44.917867 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" podUID="2ead42d5-f3f0-4e06-a578-d4d457639850" containerName="istio-proxy" containerID="cri-o://12d8b7f2df598cb5e68a435196a97ee88f3cbb29a04d91adf99e0a515587cb66" gracePeriod=30 Nov 03 17:24:45.787841 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:45.787802 4947 patch_prober.go:29] interesting pod/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:24:45.788034 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:45.787865 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" podUID="d9f0e126-f94a-4cb6-b187-e05f23476696" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:24:46.569238 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:46.569195 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7c02310-bff9-4d86-b641-8022d891b395" path="/var/lib/kubelet/pods/e7c02310-bff9-4d86-b641-8022d891b395/volumes" Nov 03 17:24:47.938268 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:47.938241 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:24:48.073775 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.073736 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-credential-socket\") pod \"d9f0e126-f94a-4cb6-b187-e05f23476696\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " Nov 03 17:24:48.073775 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.073777 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f0e126-f94a-4cb6-b187-e05f23476696-kube-api-access-g7fgv\" (UniqueName: \"kubernetes.io/projected/d9f0e126-f94a-4cb6-b187-e05f23476696-kube-api-access-g7fgv\") pod \"d9f0e126-f94a-4cb6-b187-e05f23476696\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " Nov 03 17:24:48.074057 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.073819 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-token\" (UniqueName: \"kubernetes.io/projected/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-token\") pod \"d9f0e126-f94a-4cb6-b187-e05f23476696\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " Nov 03 17:24:48.074057 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.073895 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9f0e126-f94a-4cb6-b187-e05f23476696-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f0e126-f94a-4cb6-b187-e05f23476696-istiod-ca-cert\") pod \"d9f0e126-f94a-4cb6-b187-e05f23476696\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " Nov 03 17:24:48.074057 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.073984 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-workload-socket\") pod \"d9f0e126-f94a-4cb6-b187-e05f23476696\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " Nov 03 17:24:48.074057 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.074032 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-envoy\") pod \"d9f0e126-f94a-4cb6-b187-e05f23476696\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " Nov 03 17:24:48.074288 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.074049 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-credential-socket" pod "d9f0e126-f94a-4cb6-b187-e05f23476696" (UID: "d9f0e126-f94a-4cb6-b187-e05f23476696"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:48.074288 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.074099 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-workload-certs\") pod \"d9f0e126-f94a-4cb6-b187-e05f23476696\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " Nov 03 17:24:48.074288 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.074143 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-podinfo\") pod \"d9f0e126-f94a-4cb6-b187-e05f23476696\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " Nov 03 17:24:48.074288 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.074251 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-data\") pod \"d9f0e126-f94a-4cb6-b187-e05f23476696\" (UID: \"d9f0e126-f94a-4cb6-b187-e05f23476696\") " Nov 03 17:24:48.074530 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.074292 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-workload-socket" pod "d9f0e126-f94a-4cb6-b187-e05f23476696" (UID: "d9f0e126-f94a-4cb6-b187-e05f23476696"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:48.074530 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.074313 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f0e126-f94a-4cb6-b187-e05f23476696-istiod-ca-cert" pod "d9f0e126-f94a-4cb6-b187-e05f23476696" (UID: "d9f0e126-f94a-4cb6-b187-e05f23476696"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:24:48.074530 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.074416 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-workload-certs" pod "d9f0e126-f94a-4cb6-b187-e05f23476696" (UID: "d9f0e126-f94a-4cb6-b187-e05f23476696"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:48.074869 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.074603 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-data" pod "d9f0e126-f94a-4cb6-b187-e05f23476696" (UID: "d9f0e126-f94a-4cb6-b187-e05f23476696"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:48.075226 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.075201 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:48.075322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.075233 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:48.075322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.075248 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:48.075322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.075260 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:48.075322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.075276 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9f0e126-f94a-4cb6-b187-e05f23476696-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:48.076376 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.076343 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f0e126-f94a-4cb6-b187-e05f23476696-kube-api-access-g7fgv" pod "d9f0e126-f94a-4cb6-b187-e05f23476696" (UID: "d9f0e126-f94a-4cb6-b187-e05f23476696"). InnerVolumeSpecName "kube-api-access-g7fgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:48.076376 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.076367 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-envoy" pod "d9f0e126-f94a-4cb6-b187-e05f23476696" (UID: "d9f0e126-f94a-4cb6-b187-e05f23476696"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:48.076510 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.076398 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-token" pod "d9f0e126-f94a-4cb6-b187-e05f23476696" (UID: "d9f0e126-f94a-4cb6-b187-e05f23476696"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:48.076605 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.076583 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-podinfo" pod "d9f0e126-f94a-4cb6-b187-e05f23476696" (UID: "d9f0e126-f94a-4cb6-b187-e05f23476696"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:24:48.177246 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.177153 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7fgv\" (UniqueName: \"kubernetes.io/projected/d9f0e126-f94a-4cb6-b187-e05f23476696-kube-api-access-g7fgv\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:48.177246 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.177189 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:48.177246 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.177204 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:48.177246 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.177217 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9f0e126-f94a-4cb6-b187-e05f23476696-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:48.724200 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.724167 4947 generic.go:356] "Generic (PLEG): container finished" podID="d9f0e126-f94a-4cb6-b187-e05f23476696" containerID="3dff21d654da0f2332a9f2530d080428f0be198405c87d22cfb457c427e2b694" exitCode=0 Nov 03 17:24:48.724429 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.724215 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" Nov 03 17:24:48.724429 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.724275 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" event={"ID":"d9f0e126-f94a-4cb6-b187-e05f23476696","Type":"ContainerDied","Data":"3dff21d654da0f2332a9f2530d080428f0be198405c87d22cfb457c427e2b694"} Nov 03 17:24:48.724429 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.724334 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr" event={"ID":"d9f0e126-f94a-4cb6-b187-e05f23476696","Type":"ContainerDied","Data":"c3f965f77ed836e360dab3f466d9639484acdd7df2d2e37d66886c82c4adb838"} Nov 03 17:24:48.724429 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.724363 4947 scope.go:122] "RemoveContainer" containerID="3dff21d654da0f2332a9f2530d080428f0be198405c87d22cfb457c427e2b694" Nov 03 17:24:48.736295 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.736275 4947 scope.go:122] "RemoveContainer" containerID="3dff21d654da0f2332a9f2530d080428f0be198405c87d22cfb457c427e2b694" Nov 03 17:24:48.736559 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:24:48.736541 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dff21d654da0f2332a9f2530d080428f0be198405c87d22cfb457c427e2b694\": container with ID starting with 3dff21d654da0f2332a9f2530d080428f0be198405c87d22cfb457c427e2b694 not found: ID does not exist" containerID="3dff21d654da0f2332a9f2530d080428f0be198405c87d22cfb457c427e2b694" Nov 03 17:24:48.736621 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.736567 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dff21d654da0f2332a9f2530d080428f0be198405c87d22cfb457c427e2b694"} err="failed to get container status \"3dff21d654da0f2332a9f2530d080428f0be198405c87d22cfb457c427e2b694\": rpc error: code = NotFound desc = could not find container \"3dff21d654da0f2332a9f2530d080428f0be198405c87d22cfb457c427e2b694\": container with ID starting with 3dff21d654da0f2332a9f2530d080428f0be198405c87d22cfb457c427e2b694 not found: ID does not exist" Nov 03 17:24:48.797529 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.797499 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr"] Nov 03 17:24:48.810218 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.810195 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1f83c0-openshift-default-54fc97cfb5-zfgtr"] Nov 03 17:24:48.860759 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:48.860669 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" podUID="b2938160-d944-4c19-abd6-f634b57b920b" containerName="payload-pre-processing" containerID="cri-o://291e76ab329beebdc76ff9a1ac2d0461bac5383b9878ac36b66a0ce99e822047" gracePeriod=25 Nov 03 17:24:49.511622 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:49.511586 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:24:49.512038 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:49.511649 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" podUID="2ead42d5-f3f0-4e06-a578-d4d457639850" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:24:50.215355 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.215320 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:24:50.327425 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.327368 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ead42d5-f3f0-4e06-a578-d4d457639850-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ead42d5-f3f0-4e06-a578-d4d457639850-istiod-ca-cert\") pod \"2ead42d5-f3f0-4e06-a578-d4d457639850\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " Nov 03 17:24:50.327644 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.327477 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-data\") pod \"2ead42d5-f3f0-4e06-a578-d4d457639850\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " Nov 03 17:24:50.327644 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.327550 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-workload-socket\") pod \"2ead42d5-f3f0-4e06-a578-d4d457639850\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " Nov 03 17:24:50.327644 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.327586 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-token\" (UniqueName: \"kubernetes.io/projected/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-token\") pod \"2ead42d5-f3f0-4e06-a578-d4d457639850\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " Nov 03 17:24:50.327644 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.327620 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-workload-certs\") pod \"2ead42d5-f3f0-4e06-a578-d4d457639850\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " Nov 03 17:24:50.327924 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.327768 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-podinfo\") pod \"2ead42d5-f3f0-4e06-a578-d4d457639850\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " Nov 03 17:24:50.327924 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.327792 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ead42d5-f3f0-4e06-a578-d4d457639850-istiod-ca-cert" pod "2ead42d5-f3f0-4e06-a578-d4d457639850" (UID: "2ead42d5-f3f0-4e06-a578-d4d457639850"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:24:50.327924 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.327814 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-credential-socket\") pod \"2ead42d5-f3f0-4e06-a578-d4d457639850\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " Nov 03 17:24:50.328102 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.327987 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-envoy\") pod \"2ead42d5-f3f0-4e06-a578-d4d457639850\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " Nov 03 17:24:50.328102 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.328073 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ead42d5-f3f0-4e06-a578-d4d457639850-kube-api-access-gqsvv\" (UniqueName: \"kubernetes.io/projected/2ead42d5-f3f0-4e06-a578-d4d457639850-kube-api-access-gqsvv\") pod \"2ead42d5-f3f0-4e06-a578-d4d457639850\" (UID: \"2ead42d5-f3f0-4e06-a578-d4d457639850\") " Nov 03 17:24:50.328806 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.328464 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-workload-socket" pod "2ead42d5-f3f0-4e06-a578-d4d457639850" (UID: "2ead42d5-f3f0-4e06-a578-d4d457639850"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:50.328806 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.328583 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-credential-socket" pod "2ead42d5-f3f0-4e06-a578-d4d457639850" (UID: "2ead42d5-f3f0-4e06-a578-d4d457639850"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:50.328806 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.328661 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-workload-certs" pod "2ead42d5-f3f0-4e06-a578-d4d457639850" (UID: "2ead42d5-f3f0-4e06-a578-d4d457639850"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:50.329146 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.329121 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:50.329235 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.329153 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:50.329346 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.329168 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:50.329431 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.329359 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ead42d5-f3f0-4e06-a578-d4d457639850-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:50.329625 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.329596 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-data" pod "2ead42d5-f3f0-4e06-a578-d4d457639850" (UID: "2ead42d5-f3f0-4e06-a578-d4d457639850"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:50.332225 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.332197 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ead42d5-f3f0-4e06-a578-d4d457639850-kube-api-access-gqsvv" pod "2ead42d5-f3f0-4e06-a578-d4d457639850" (UID: "2ead42d5-f3f0-4e06-a578-d4d457639850"). InnerVolumeSpecName "kube-api-access-gqsvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:50.332473 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.332446 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-token" pod "2ead42d5-f3f0-4e06-a578-d4d457639850" (UID: "2ead42d5-f3f0-4e06-a578-d4d457639850"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:24:50.335619 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.335580 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-envoy" pod "2ead42d5-f3f0-4e06-a578-d4d457639850" (UID: "2ead42d5-f3f0-4e06-a578-d4d457639850"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:24:50.337274 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.337251 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-podinfo" pod "2ead42d5-f3f0-4e06-a578-d4d457639850" (UID: "2ead42d5-f3f0-4e06-a578-d4d457639850"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:24:50.431284 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.431251 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:50.431284 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.431282 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:50.431284 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.431291 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqsvv\" (UniqueName: \"kubernetes.io/projected/2ead42d5-f3f0-4e06-a578-d4d457639850-kube-api-access-gqsvv\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:50.431537 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.431300 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:50.431537 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.431309 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ead42d5-f3f0-4e06-a578-d4d457639850-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:24:50.567414 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.567379 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9f0e126-f94a-4cb6-b187-e05f23476696" path="/var/lib/kubelet/pods/d9f0e126-f94a-4cb6-b187-e05f23476696/volumes" Nov 03 17:24:50.761151 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.761105 4947 generic.go:356] "Generic (PLEG): container finished" podID="2ead42d5-f3f0-4e06-a578-d4d457639850" containerID="12d8b7f2df598cb5e68a435196a97ee88f3cbb29a04d91adf99e0a515587cb66" exitCode=0 Nov 03 17:24:50.761371 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.761206 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" event={"ID":"2ead42d5-f3f0-4e06-a578-d4d457639850","Type":"ContainerDied","Data":"12d8b7f2df598cb5e68a435196a97ee88f3cbb29a04d91adf99e0a515587cb66"} Nov 03 17:24:50.761371 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.761243 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" Nov 03 17:24:50.761371 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.761316 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst" event={"ID":"2ead42d5-f3f0-4e06-a578-d4d457639850","Type":"ContainerDied","Data":"c481069bc25fdfbe5b5d2343884a7c0f0cc3e4523632d2e4e0d2a32835c2a887"} Nov 03 17:24:50.761371 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.761349 4947 scope.go:122] "RemoveContainer" containerID="12d8b7f2df598cb5e68a435196a97ee88f3cbb29a04d91adf99e0a515587cb66" Nov 03 17:24:50.773961 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.773930 4947 scope.go:122] "RemoveContainer" containerID="12d8b7f2df598cb5e68a435196a97ee88f3cbb29a04d91adf99e0a515587cb66" Nov 03 17:24:50.774264 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:24:50.774236 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12d8b7f2df598cb5e68a435196a97ee88f3cbb29a04d91adf99e0a515587cb66\": container with ID starting with 12d8b7f2df598cb5e68a435196a97ee88f3cbb29a04d91adf99e0a515587cb66 not found: ID does not exist" containerID="12d8b7f2df598cb5e68a435196a97ee88f3cbb29a04d91adf99e0a515587cb66" Nov 03 17:24:50.774319 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.774275 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12d8b7f2df598cb5e68a435196a97ee88f3cbb29a04d91adf99e0a515587cb66"} err="failed to get container status \"12d8b7f2df598cb5e68a435196a97ee88f3cbb29a04d91adf99e0a515587cb66\": rpc error: code = NotFound desc = could not find container \"12d8b7f2df598cb5e68a435196a97ee88f3cbb29a04d91adf99e0a515587cb66\": container with ID starting with 12d8b7f2df598cb5e68a435196a97ee88f3cbb29a04d91adf99e0a515587cb66 not found: ID does not exist" Nov 03 17:24:50.829157 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.829089 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst"] Nov 03 17:24:50.846500 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:50.846473 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-ca9d7c-openshift-default-6fb79bb6df-qtgst"] Nov 03 17:24:52.028593 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:52.028560 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" podUID="b2938160-d944-4c19-abd6-f634b57b920b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:24:52.571650 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:24:52.571618 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ead42d5-f3f0-4e06-a578-d4d457639850" path="/var/lib/kubelet/pods/2ead42d5-f3f0-4e06-a578-d4d457639850/volumes" Nov 03 17:25:00.131007 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:25:00.130965 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:25:02.027811 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:02.027772 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" podUID="b2938160-d944-4c19-abd6-f634b57b920b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:25:03.483407 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:03.483373 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb"] Nov 03 17:25:03.483899 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:03.483628 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" podUID="0fc48229-09ca-4ae5-9cb2-de8e8ea21cad" containerName="maas-api" containerID="cri-o://2a61e72958c4d631f5ac37bc753c2fe6acfab3bfce2068a1893f18c880afb9bb" gracePeriod=30 Nov 03 17:25:03.892824 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:03.892797 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:25:04.032725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.025591 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-metrics-certs\") pod \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\" (UID: \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\") " Nov 03 17:25:04.032725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.025673 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-kube-api-access-hxqcb\" (UniqueName: \"kubernetes.io/projected/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-kube-api-access-hxqcb\") pod \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\" (UID: \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\") " Nov 03 17:25:04.032725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.029840 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-maas-api-tls\") pod \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\" (UID: \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\") " Nov 03 17:25:04.032725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.029927 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-openshift-service-ca\") pod \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\" (UID: \"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad\") " Nov 03 17:25:04.056727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.053910 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-openshift-service-ca" pod "0fc48229-09ca-4ae5-9cb2-de8e8ea21cad" (UID: "0fc48229-09ca-4ae5-9cb2-de8e8ea21cad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:04.104721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.102514 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-metrics-certs" pod "0fc48229-09ca-4ae5-9cb2-de8e8ea21cad" (UID: "0fc48229-09ca-4ae5-9cb2-de8e8ea21cad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:25:04.151757 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.144059 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:04.151757 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.144097 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:04.151757 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.147280 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-kube-api-access-hxqcb" pod "0fc48229-09ca-4ae5-9cb2-de8e8ea21cad" (UID: "0fc48229-09ca-4ae5-9cb2-de8e8ea21cad"). InnerVolumeSpecName "kube-api-access-hxqcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:04.151757 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.147777 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-maas-api-tls" pod "0fc48229-09ca-4ae5-9cb2-de8e8ea21cad" (UID: "0fc48229-09ca-4ae5-9cb2-de8e8ea21cad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:25:04.151757 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.149646 4947 generic.go:356] "Generic (PLEG): container finished" podID="0fc48229-09ca-4ae5-9cb2-de8e8ea21cad" containerID="2a61e72958c4d631f5ac37bc753c2fe6acfab3bfce2068a1893f18c880afb9bb" exitCode=0 Nov 03 17:25:04.151757 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.149840 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" Nov 03 17:25:04.151757 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.149945 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" event={"ID":"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad","Type":"ContainerDied","Data":"2a61e72958c4d631f5ac37bc753c2fe6acfab3bfce2068a1893f18c880afb9bb"} Nov 03 17:25:04.151757 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.149994 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb" event={"ID":"0fc48229-09ca-4ae5-9cb2-de8e8ea21cad","Type":"ContainerDied","Data":"ee3aacf78c2bdb4e2c05f67eab089a8e10bc41557772463465875bb459d5f4e1"} Nov 03 17:25:04.151757 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.150023 4947 scope.go:122] "RemoveContainer" containerID="2a61e72958c4d631f5ac37bc753c2fe6acfab3bfce2068a1893f18c880afb9bb" Nov 03 17:25:04.179498 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.179472 4947 scope.go:122] "RemoveContainer" containerID="2a61e72958c4d631f5ac37bc753c2fe6acfab3bfce2068a1893f18c880afb9bb" Nov 03 17:25:04.179888 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:25:04.179861 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a61e72958c4d631f5ac37bc753c2fe6acfab3bfce2068a1893f18c880afb9bb\": container with ID starting with 2a61e72958c4d631f5ac37bc753c2fe6acfab3bfce2068a1893f18c880afb9bb not found: ID does not exist" containerID="2a61e72958c4d631f5ac37bc753c2fe6acfab3bfce2068a1893f18c880afb9bb" Nov 03 17:25:04.180029 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.179894 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a61e72958c4d631f5ac37bc753c2fe6acfab3bfce2068a1893f18c880afb9bb"} err="failed to get container status \"2a61e72958c4d631f5ac37bc753c2fe6acfab3bfce2068a1893f18c880afb9bb\": rpc error: code = NotFound desc = could not find container \"2a61e72958c4d631f5ac37bc753c2fe6acfab3bfce2068a1893f18c880afb9bb\": container with ID starting with 2a61e72958c4d631f5ac37bc753c2fe6acfab3bfce2068a1893f18c880afb9bb not found: ID does not exist" Nov 03 17:25:04.214881 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.214853 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb"] Nov 03 17:25:04.218638 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.218602 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-24cb6d-574c99f4c-fm9sb"] Nov 03 17:25:04.245622 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.245595 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxqcb\" (UniqueName: \"kubernetes.io/projected/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-kube-api-access-hxqcb\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:04.245622 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.245620 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:04.567480 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:04.567435 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc48229-09ca-4ae5-9cb2-de8e8ea21cad" path="/var/lib/kubelet/pods/0fc48229-09ca-4ae5-9cb2-de8e8ea21cad/volumes" Nov 03 17:25:09.151742 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:09.151717 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" Nov 03 17:25:09.191725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:09.191664 4947 generic.go:356] "Generic (PLEG): container finished" podID="b2938160-d944-4c19-abd6-f634b57b920b" containerID="291e76ab329beebdc76ff9a1ac2d0461bac5383b9878ac36b66a0ce99e822047" exitCode=0 Nov 03 17:25:09.191961 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:09.191780 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" Nov 03 17:25:09.192043 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:09.191838 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" event={"ID":"b2938160-d944-4c19-abd6-f634b57b920b","Type":"ContainerDied","Data":"291e76ab329beebdc76ff9a1ac2d0461bac5383b9878ac36b66a0ce99e822047"} Nov 03 17:25:09.192108 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:09.192068 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt" event={"ID":"b2938160-d944-4c19-abd6-f634b57b920b","Type":"ContainerDied","Data":"ab4f3026e296b7e95d8f7d57716bd46d3c889668e6be638d9d1f93a3b09ea9cc"} Nov 03 17:25:09.192108 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:09.192098 4947 scope.go:122] "RemoveContainer" containerID="291e76ab329beebdc76ff9a1ac2d0461bac5383b9878ac36b66a0ce99e822047" Nov 03 17:25:09.208557 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:09.208538 4947 scope.go:122] "RemoveContainer" containerID="291e76ab329beebdc76ff9a1ac2d0461bac5383b9878ac36b66a0ce99e822047" Nov 03 17:25:09.208922 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:25:09.208876 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291e76ab329beebdc76ff9a1ac2d0461bac5383b9878ac36b66a0ce99e822047\": container with ID starting with 291e76ab329beebdc76ff9a1ac2d0461bac5383b9878ac36b66a0ce99e822047 not found: ID does not exist" containerID="291e76ab329beebdc76ff9a1ac2d0461bac5383b9878ac36b66a0ce99e822047" Nov 03 17:25:09.209040 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:09.208914 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291e76ab329beebdc76ff9a1ac2d0461bac5383b9878ac36b66a0ce99e822047"} err="failed to get container status \"291e76ab329beebdc76ff9a1ac2d0461bac5383b9878ac36b66a0ce99e822047\": rpc error: code = NotFound desc = could not find container \"291e76ab329beebdc76ff9a1ac2d0461bac5383b9878ac36b66a0ce99e822047\": container with ID starting with 291e76ab329beebdc76ff9a1ac2d0461bac5383b9878ac36b66a0ce99e822047 not found: ID does not exist" Nov 03 17:25:09.219481 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:09.219452 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2938160-d944-4c19-abd6-f634b57b920b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2938160-d944-4c19-abd6-f634b57b920b-plugins-config-volume\") pod \"b2938160-d944-4c19-abd6-f634b57b920b\" (UID: \"b2938160-d944-4c19-abd6-f634b57b920b\") " Nov 03 17:25:09.219598 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:09.219543 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2938160-d944-4c19-abd6-f634b57b920b-kube-api-access-kknlz\" (UniqueName: \"kubernetes.io/projected/b2938160-d944-4c19-abd6-f634b57b920b-kube-api-access-kknlz\") pod \"b2938160-d944-4c19-abd6-f634b57b920b\" (UID: \"b2938160-d944-4c19-abd6-f634b57b920b\") " Nov 03 17:25:09.219950 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:09.219923 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2938160-d944-4c19-abd6-f634b57b920b-plugins-config-volume" pod "b2938160-d944-4c19-abd6-f634b57b920b" (UID: "b2938160-d944-4c19-abd6-f634b57b920b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:09.220177 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:09.220155 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2938160-d944-4c19-abd6-f634b57b920b-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:09.237919 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:09.237897 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2938160-d944-4c19-abd6-f634b57b920b-kube-api-access-kknlz" pod "b2938160-d944-4c19-abd6-f634b57b920b" (UID: "b2938160-d944-4c19-abd6-f634b57b920b"). InnerVolumeSpecName "kube-api-access-kknlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:09.322298 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:09.322262 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kknlz\" (UniqueName: \"kubernetes.io/projected/b2938160-d944-4c19-abd6-f634b57b920b-kube-api-access-kknlz\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:09.547317 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:09.547285 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt"] Nov 03 17:25:09.560723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:09.551651 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-9c4762-54956bdcd7-9kjdt"] Nov 03 17:25:10.481833 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:10.481802 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77"] Nov 03 17:25:10.482244 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:10.482026 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" podUID="a852b387-ca23-4e7a-b793-788feb393176" containerName="istio-proxy" containerID="cri-o://6a9acb0c8e315371c88e962a89df8c015e929ad4cc049118a6fa34459283f8b6" gracePeriod=30 Nov 03 17:25:10.571499 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:10.571464 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2938160-d944-4c19-abd6-f634b57b920b" path="/var/lib/kubelet/pods/b2938160-d944-4c19-abd6-f634b57b920b/volumes" Nov 03 17:25:13.899210 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:13.899173 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp"] Nov 03 17:25:15.773235 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.773207 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:25:15.855581 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.855544 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a852b387-ca23-4e7a-b793-788feb393176-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a852b387-ca23-4e7a-b793-788feb393176-istiod-ca-cert\") pod \"a852b387-ca23-4e7a-b793-788feb393176\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " Nov 03 17:25:15.855784 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.855598 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-istio-envoy\") pod \"a852b387-ca23-4e7a-b793-788feb393176\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " Nov 03 17:25:15.855784 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.855659 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a852b387-ca23-4e7a-b793-788feb393176-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a852b387-ca23-4e7a-b793-788feb393176-istio-podinfo\") pod \"a852b387-ca23-4e7a-b793-788feb393176\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " Nov 03 17:25:15.855927 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.855804 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-workload-certs\") pod \"a852b387-ca23-4e7a-b793-788feb393176\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " Nov 03 17:25:15.855927 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.855863 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a852b387-ca23-4e7a-b793-788feb393176-istio-token\" (UniqueName: \"kubernetes.io/projected/a852b387-ca23-4e7a-b793-788feb393176-istio-token\") pod \"a852b387-ca23-4e7a-b793-788feb393176\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " Nov 03 17:25:15.855927 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.855902 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-workload-socket\") pod \"a852b387-ca23-4e7a-b793-788feb393176\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " Nov 03 17:25:15.856100 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.855948 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a852b387-ca23-4e7a-b793-788feb393176-kube-api-access-slsww\" (UniqueName: \"kubernetes.io/projected/a852b387-ca23-4e7a-b793-788feb393176-kube-api-access-slsww\") pod \"a852b387-ca23-4e7a-b793-788feb393176\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " Nov 03 17:25:15.856100 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.855976 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-credential-socket\") pod \"a852b387-ca23-4e7a-b793-788feb393176\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " Nov 03 17:25:15.856100 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.856015 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a852b387-ca23-4e7a-b793-788feb393176-istiod-ca-cert" pod "a852b387-ca23-4e7a-b793-788feb393176" (UID: "a852b387-ca23-4e7a-b793-788feb393176"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:15.857330 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.856251 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-istio-data\") pod \"a852b387-ca23-4e7a-b793-788feb393176\" (UID: \"a852b387-ca23-4e7a-b793-788feb393176\") " Nov 03 17:25:15.857330 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.856758 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-workload-certs" pod "a852b387-ca23-4e7a-b793-788feb393176" (UID: "a852b387-ca23-4e7a-b793-788feb393176"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:15.857330 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.856965 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-istio-data" pod "a852b387-ca23-4e7a-b793-788feb393176" (UID: "a852b387-ca23-4e7a-b793-788feb393176"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:15.857330 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.857047 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-credential-socket" pod "a852b387-ca23-4e7a-b793-788feb393176" (UID: "a852b387-ca23-4e7a-b793-788feb393176"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:15.857330 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.857283 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a852b387-ca23-4e7a-b793-788feb393176-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:15.857330 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.857303 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:15.858511 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.858474 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-workload-socket" pod "a852b387-ca23-4e7a-b793-788feb393176" (UID: "a852b387-ca23-4e7a-b793-788feb393176"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:15.859490 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.859456 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-istio-envoy" pod "a852b387-ca23-4e7a-b793-788feb393176" (UID: "a852b387-ca23-4e7a-b793-788feb393176"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:15.859592 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.859532 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a852b387-ca23-4e7a-b793-788feb393176-kube-api-access-slsww" pod "a852b387-ca23-4e7a-b793-788feb393176" (UID: "a852b387-ca23-4e7a-b793-788feb393176"). InnerVolumeSpecName "kube-api-access-slsww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:15.860502 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.860480 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a852b387-ca23-4e7a-b793-788feb393176-istio-podinfo" pod "a852b387-ca23-4e7a-b793-788feb393176" (UID: "a852b387-ca23-4e7a-b793-788feb393176"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:25:15.860825 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.860803 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a852b387-ca23-4e7a-b793-788feb393176-istio-token" pod "a852b387-ca23-4e7a-b793-788feb393176" (UID: "a852b387-ca23-4e7a-b793-788feb393176"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:15.958312 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.958280 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:15.958312 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.958306 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:15.958312 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.958316 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a852b387-ca23-4e7a-b793-788feb393176-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:15.958556 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.958325 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a852b387-ca23-4e7a-b793-788feb393176-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:15.958556 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.958332 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:15.958556 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.958342 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slsww\" (UniqueName: \"kubernetes.io/projected/a852b387-ca23-4e7a-b793-788feb393176-kube-api-access-slsww\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:15.958556 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:15.958351 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a852b387-ca23-4e7a-b793-788feb393176-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:16.260958 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:16.260923 4947 generic.go:356] "Generic (PLEG): container finished" podID="a852b387-ca23-4e7a-b793-788feb393176" containerID="6a9acb0c8e315371c88e962a89df8c015e929ad4cc049118a6fa34459283f8b6" exitCode=0 Nov 03 17:25:16.261152 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:16.260974 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" Nov 03 17:25:16.261152 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:16.261003 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" event={"ID":"a852b387-ca23-4e7a-b793-788feb393176","Type":"ContainerDied","Data":"6a9acb0c8e315371c88e962a89df8c015e929ad4cc049118a6fa34459283f8b6"} Nov 03 17:25:16.261152 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:16.261057 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77" event={"ID":"a852b387-ca23-4e7a-b793-788feb393176","Type":"ContainerDied","Data":"902e7002ca49960b485b366c98ff554ac40e77a6b07505210b3a16b8b75a5474"} Nov 03 17:25:16.261152 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:16.261078 4947 scope.go:122] "RemoveContainer" containerID="6a9acb0c8e315371c88e962a89df8c015e929ad4cc049118a6fa34459283f8b6" Nov 03 17:25:16.275757 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:16.275734 4947 scope.go:122] "RemoveContainer" containerID="137cafd0e2519e9b29e6b9d0894445a8fd77e6b3aa47de07214a3dd16c740ae5" Nov 03 17:25:16.288662 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:16.288638 4947 scope.go:122] "RemoveContainer" containerID="6a9acb0c8e315371c88e962a89df8c015e929ad4cc049118a6fa34459283f8b6" Nov 03 17:25:16.288962 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:25:16.288939 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a9acb0c8e315371c88e962a89df8c015e929ad4cc049118a6fa34459283f8b6\": container with ID starting with 6a9acb0c8e315371c88e962a89df8c015e929ad4cc049118a6fa34459283f8b6 not found: ID does not exist" containerID="6a9acb0c8e315371c88e962a89df8c015e929ad4cc049118a6fa34459283f8b6" Nov 03 17:25:16.289020 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:16.288972 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a9acb0c8e315371c88e962a89df8c015e929ad4cc049118a6fa34459283f8b6"} err="failed to get container status \"6a9acb0c8e315371c88e962a89df8c015e929ad4cc049118a6fa34459283f8b6\": rpc error: code = NotFound desc = could not find container \"6a9acb0c8e315371c88e962a89df8c015e929ad4cc049118a6fa34459283f8b6\": container with ID starting with 6a9acb0c8e315371c88e962a89df8c015e929ad4cc049118a6fa34459283f8b6 not found: ID does not exist" Nov 03 17:25:16.289020 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:16.288989 4947 scope.go:122] "RemoveContainer" containerID="137cafd0e2519e9b29e6b9d0894445a8fd77e6b3aa47de07214a3dd16c740ae5" Nov 03 17:25:16.289242 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:25:16.289216 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"137cafd0e2519e9b29e6b9d0894445a8fd77e6b3aa47de07214a3dd16c740ae5\": container with ID starting with 137cafd0e2519e9b29e6b9d0894445a8fd77e6b3aa47de07214a3dd16c740ae5 not found: ID does not exist" containerID="137cafd0e2519e9b29e6b9d0894445a8fd77e6b3aa47de07214a3dd16c740ae5" Nov 03 17:25:16.289342 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:16.289248 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"137cafd0e2519e9b29e6b9d0894445a8fd77e6b3aa47de07214a3dd16c740ae5"} err="failed to get container status \"137cafd0e2519e9b29e6b9d0894445a8fd77e6b3aa47de07214a3dd16c740ae5\": rpc error: code = NotFound desc = could not find container \"137cafd0e2519e9b29e6b9d0894445a8fd77e6b3aa47de07214a3dd16c740ae5\": container with ID starting with 137cafd0e2519e9b29e6b9d0894445a8fd77e6b3aa47de07214a3dd16c740ae5 not found: ID does not exist" Nov 03 17:25:16.323299 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:16.323273 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77"] Nov 03 17:25:16.330500 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:16.330476 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-24cb6d-openshift-default-84c588c649-7dl77"] Nov 03 17:25:16.568366 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:16.568293 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a852b387-ca23-4e7a-b793-788feb393176" path="/var/lib/kubelet/pods/a852b387-ca23-4e7a-b793-788feb393176/volumes" Nov 03 17:25:28.929533 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:28.929447 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" podUID="2d4fef6b-5ca6-41c1-8809-a74e3c3401e1" containerName="main" containerID="cri-o://eabbad8f9c5b027ead65747991cebb3bb2d6a910c6797b7ba601fc4c33935cee" gracePeriod=45 Nov 03 17:25:29.226033 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.226006 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:25:29.296799 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.296558 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-home\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-home\") pod \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " Nov 03 17:25:29.296799 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.296676 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-kube-api-access-7xwtw\" (UniqueName: \"kubernetes.io/projected/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-kube-api-access-7xwtw\") pod \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " Nov 03 17:25:29.296799 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.296785 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-tls-certs\") pod \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " Nov 03 17:25:29.297086 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.296866 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-tmp-dir\") pod \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " Nov 03 17:25:29.297086 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.296993 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-dshm\") pod \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " Nov 03 17:25:29.297204 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.297126 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-model-cache\") pod \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\" (UID: \"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1\") " Nov 03 17:25:29.297345 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.297311 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-home" pod "2d4fef6b-5ca6-41c1-8809-a74e3c3401e1" (UID: "2d4fef6b-5ca6-41c1-8809-a74e3c3401e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:29.297875 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.297855 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-tmp-dir" pod "2d4fef6b-5ca6-41c1-8809-a74e3c3401e1" (UID: "2d4fef6b-5ca6-41c1-8809-a74e3c3401e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:29.298018 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.297971 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-model-cache" pod "2d4fef6b-5ca6-41c1-8809-a74e3c3401e1" (UID: "2d4fef6b-5ca6-41c1-8809-a74e3c3401e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:29.298129 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.297972 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:29.300546 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.300516 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-tls-certs" pod "2d4fef6b-5ca6-41c1-8809-a74e3c3401e1" (UID: "2d4fef6b-5ca6-41c1-8809-a74e3c3401e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:25:29.300686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.300590 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-dshm" pod "2d4fef6b-5ca6-41c1-8809-a74e3c3401e1" (UID: "2d4fef6b-5ca6-41c1-8809-a74e3c3401e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:29.302054 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.302030 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-kube-api-access-7xwtw" pod "2d4fef6b-5ca6-41c1-8809-a74e3c3401e1" (UID: "2d4fef6b-5ca6-41c1-8809-a74e3c3401e1"). InnerVolumeSpecName "kube-api-access-7xwtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:29.406972 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.401172 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:29.406972 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.401208 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:29.406972 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.401223 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:29.406972 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.401247 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xwtw\" (UniqueName: \"kubernetes.io/projected/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-kube-api-access-7xwtw\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:29.406972 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.401260 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:29.484616 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.484530 4947 generic.go:356] "Generic (PLEG): container finished" podID="2d4fef6b-5ca6-41c1-8809-a74e3c3401e1" containerID="eabbad8f9c5b027ead65747991cebb3bb2d6a910c6797b7ba601fc4c33935cee" exitCode=0 Nov 03 17:25:29.484616 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.484593 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" Nov 03 17:25:29.484904 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.484611 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" event={"ID":"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1","Type":"ContainerDied","Data":"eabbad8f9c5b027ead65747991cebb3bb2d6a910c6797b7ba601fc4c33935cee"} Nov 03 17:25:29.484904 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.484653 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" event={"ID":"2d4fef6b-5ca6-41c1-8809-a74e3c3401e1","Type":"ContainerDied","Data":"9b571f290c3068768c2b69a5995292b51c20bc99ac19957816d7d2d2cba9d815"} Nov 03 17:25:29.484904 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.484670 4947 scope.go:122] "RemoveContainer" containerID="eabbad8f9c5b027ead65747991cebb3bb2d6a910c6797b7ba601fc4c33935cee" Nov 03 17:25:29.495914 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.495894 4947 scope.go:122] "RemoveContainer" containerID="eabbad8f9c5b027ead65747991cebb3bb2d6a910c6797b7ba601fc4c33935cee" Nov 03 17:25:29.496209 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:25:29.496193 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eabbad8f9c5b027ead65747991cebb3bb2d6a910c6797b7ba601fc4c33935cee\": container with ID starting with eabbad8f9c5b027ead65747991cebb3bb2d6a910c6797b7ba601fc4c33935cee not found: ID does not exist" containerID="eabbad8f9c5b027ead65747991cebb3bb2d6a910c6797b7ba601fc4c33935cee" Nov 03 17:25:29.496262 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.496218 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eabbad8f9c5b027ead65747991cebb3bb2d6a910c6797b7ba601fc4c33935cee"} err="failed to get container status \"eabbad8f9c5b027ead65747991cebb3bb2d6a910c6797b7ba601fc4c33935cee\": rpc error: code = NotFound desc = could not find container \"eabbad8f9c5b027ead65747991cebb3bb2d6a910c6797b7ba601fc4c33935cee\": container with ID starting with eabbad8f9c5b027ead65747991cebb3bb2d6a910c6797b7ba601fc4c33935cee not found: ID does not exist" Nov 03 17:25:29.543951 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.543921 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp"] Nov 03 17:25:29.547818 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.547790 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp"] Nov 03 17:25:29.897973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:29.897934 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2"] Nov 03 17:25:30.190178 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:25:30.190017 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:25:30.214761 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:30.214686 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-cb0fa3/test-model-cb0fa3-kserve-5b4ddcbcc-kmbvp" podUID="2d4fef6b-5ca6-41c1-8809-a74e3c3401e1" containerName="main" probeResult="failure" output="Get \"https://10.132.1.9:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:25:30.567882 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:30.567847 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d4fef6b-5ca6-41c1-8809-a74e3c3401e1" path="/var/lib/kubelet/pods/2d4fef6b-5ca6-41c1-8809-a74e3c3401e1/volumes" Nov 03 17:25:32.989498 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:32.989456 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv"] Nov 03 17:25:32.990074 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:32.989671 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" podUID="1513cf28-6652-4788-a4eb-215a9cf87a77" containerName="maas-api" containerID="cri-o://3b978ff48dfb0627b9652e306545746182f1c0362f7c60da83b6e17465b63e68" gracePeriod=30 Nov 03 17:25:33.521640 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.521274 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" Nov 03 17:25:33.546507 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.546469 4947 generic.go:356] "Generic (PLEG): container finished" podID="1513cf28-6652-4788-a4eb-215a9cf87a77" containerID="3b978ff48dfb0627b9652e306545746182f1c0362f7c60da83b6e17465b63e68" exitCode=0 Nov 03 17:25:33.546763 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.546524 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" event={"ID":"1513cf28-6652-4788-a4eb-215a9cf87a77","Type":"ContainerDied","Data":"3b978ff48dfb0627b9652e306545746182f1c0362f7c60da83b6e17465b63e68"} Nov 03 17:25:33.546763 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.546565 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" event={"ID":"1513cf28-6652-4788-a4eb-215a9cf87a77","Type":"ContainerDied","Data":"1facd65e18bd691eeb59916d36dc0d1c8c1dbe8595f958523d1cdfac6951737f"} Nov 03 17:25:33.546763 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.546587 4947 scope.go:122] "RemoveContainer" containerID="3b978ff48dfb0627b9652e306545746182f1c0362f7c60da83b6e17465b63e68" Nov 03 17:25:33.546763 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.546681 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv" Nov 03 17:25:33.569767 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.569433 4947 scope.go:122] "RemoveContainer" containerID="3b978ff48dfb0627b9652e306545746182f1c0362f7c60da83b6e17465b63e68" Nov 03 17:25:33.624409 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:25:33.624320 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b978ff48dfb0627b9652e306545746182f1c0362f7c60da83b6e17465b63e68\": container with ID starting with 3b978ff48dfb0627b9652e306545746182f1c0362f7c60da83b6e17465b63e68 not found: ID does not exist" containerID="3b978ff48dfb0627b9652e306545746182f1c0362f7c60da83b6e17465b63e68" Nov 03 17:25:33.624409 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.624369 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b978ff48dfb0627b9652e306545746182f1c0362f7c60da83b6e17465b63e68"} err="failed to get container status \"3b978ff48dfb0627b9652e306545746182f1c0362f7c60da83b6e17465b63e68\": rpc error: code = NotFound desc = could not find container \"3b978ff48dfb0627b9652e306545746182f1c0362f7c60da83b6e17465b63e68\": container with ID starting with 3b978ff48dfb0627b9652e306545746182f1c0362f7c60da83b6e17465b63e68 not found: ID does not exist" Nov 03 17:25:33.632595 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.632561 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs"] Nov 03 17:25:33.632911 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.632884 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" podUID="cfcceda1-486c-48fe-a25e-d7d37f97e5a3" containerName="istio-proxy" containerID="cri-o://d6e8d70153e827d4e5eeb57bf67a7272f2a0ef232bcb615d65c6c5e37c9ac3d8" gracePeriod=30 Nov 03 17:25:33.702722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.701726 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1513cf28-6652-4788-a4eb-215a9cf87a77-kube-api-access-wwspx\" (UniqueName: \"kubernetes.io/projected/1513cf28-6652-4788-a4eb-215a9cf87a77-kube-api-access-wwspx\") pod \"1513cf28-6652-4788-a4eb-215a9cf87a77\" (UID: \"1513cf28-6652-4788-a4eb-215a9cf87a77\") " Nov 03 17:25:33.702722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.701858 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1513cf28-6652-4788-a4eb-215a9cf87a77-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1513cf28-6652-4788-a4eb-215a9cf87a77-metrics-certs\") pod \"1513cf28-6652-4788-a4eb-215a9cf87a77\" (UID: \"1513cf28-6652-4788-a4eb-215a9cf87a77\") " Nov 03 17:25:33.702722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.702000 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1513cf28-6652-4788-a4eb-215a9cf87a77-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1513cf28-6652-4788-a4eb-215a9cf87a77-maas-api-tls\") pod \"1513cf28-6652-4788-a4eb-215a9cf87a77\" (UID: \"1513cf28-6652-4788-a4eb-215a9cf87a77\") " Nov 03 17:25:33.702722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.702202 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1513cf28-6652-4788-a4eb-215a9cf87a77-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1513cf28-6652-4788-a4eb-215a9cf87a77-openshift-service-ca\") pod \"1513cf28-6652-4788-a4eb-215a9cf87a77\" (UID: \"1513cf28-6652-4788-a4eb-215a9cf87a77\") " Nov 03 17:25:33.703106 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.702937 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1513cf28-6652-4788-a4eb-215a9cf87a77-openshift-service-ca" pod "1513cf28-6652-4788-a4eb-215a9cf87a77" (UID: "1513cf28-6652-4788-a4eb-215a9cf87a77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:33.705989 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.705949 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1513cf28-6652-4788-a4eb-215a9cf87a77-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:33.707821 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.707791 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1513cf28-6652-4788-a4eb-215a9cf87a77-kube-api-access-wwspx" pod "1513cf28-6652-4788-a4eb-215a9cf87a77" (UID: "1513cf28-6652-4788-a4eb-215a9cf87a77"). InnerVolumeSpecName "kube-api-access-wwspx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:33.708239 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.708215 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1513cf28-6652-4788-a4eb-215a9cf87a77-maas-api-tls" pod "1513cf28-6652-4788-a4eb-215a9cf87a77" (UID: "1513cf28-6652-4788-a4eb-215a9cf87a77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:25:33.708903 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.708876 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1513cf28-6652-4788-a4eb-215a9cf87a77-metrics-certs" pod "1513cf28-6652-4788-a4eb-215a9cf87a77" (UID: "1513cf28-6652-4788-a4eb-215a9cf87a77"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:25:33.807184 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.807151 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwspx\" (UniqueName: \"kubernetes.io/projected/1513cf28-6652-4788-a4eb-215a9cf87a77-kube-api-access-wwspx\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:33.807184 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.807180 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1513cf28-6652-4788-a4eb-215a9cf87a77-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:33.807184 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.807189 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1513cf28-6652-4788-a4eb-215a9cf87a77-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:33.931977 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.931893 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv"] Nov 03 17:25:33.937434 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:33.937400 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c4762-5f8b7f9bd-6pflv"] Nov 03 17:25:34.565856 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:34.565824 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1513cf28-6652-4788-a4eb-215a9cf87a77" path="/var/lib/kubelet/pods/1513cf28-6652-4788-a4eb-215a9cf87a77/volumes" Nov 03 17:25:38.929035 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:38.929003 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:25:39.011371 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.011325 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-token\" (UniqueName: \"kubernetes.io/projected/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-token\") pod \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " Nov 03 17:25:39.011585 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.011483 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-data\") pod \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " Nov 03 17:25:39.012438 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.012404 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-data" pod "cfcceda1-486c-48fe-a25e-d7d37f97e5a3" (UID: "cfcceda1-486c-48fe-a25e-d7d37f97e5a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:39.012568 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.012472 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-kube-api-access-nqfmd\" (UniqueName: \"kubernetes.io/projected/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-kube-api-access-nqfmd\") pod \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " Nov 03 17:25:39.012568 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.012540 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-credential-socket\") pod \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " Nov 03 17:25:39.013786 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.012688 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-envoy\") pod \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " Nov 03 17:25:39.013893 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.013844 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istiod-ca-cert\") pod \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " Nov 03 17:25:39.013893 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.013878 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-workload-socket\") pod \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " Nov 03 17:25:39.014021 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.013912 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-workload-certs\") pod \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " Nov 03 17:25:39.014021 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.013960 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-podinfo\") pod \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\" (UID: \"cfcceda1-486c-48fe-a25e-d7d37f97e5a3\") " Nov 03 17:25:39.014917 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.012997 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-credential-socket" pod "cfcceda1-486c-48fe-a25e-d7d37f97e5a3" (UID: "cfcceda1-486c-48fe-a25e-d7d37f97e5a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:39.014917 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.014886 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istiod-ca-cert" pod "cfcceda1-486c-48fe-a25e-d7d37f97e5a3" (UID: "cfcceda1-486c-48fe-a25e-d7d37f97e5a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:39.015170 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.015147 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-workload-certs" pod "cfcceda1-486c-48fe-a25e-d7d37f97e5a3" (UID: "cfcceda1-486c-48fe-a25e-d7d37f97e5a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:39.015242 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.015185 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-workload-socket" pod "cfcceda1-486c-48fe-a25e-d7d37f97e5a3" (UID: "cfcceda1-486c-48fe-a25e-d7d37f97e5a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:39.015242 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.015234 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:39.017580 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.017547 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-kube-api-access-nqfmd" pod "cfcceda1-486c-48fe-a25e-d7d37f97e5a3" (UID: "cfcceda1-486c-48fe-a25e-d7d37f97e5a3"). InnerVolumeSpecName "kube-api-access-nqfmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:39.018358 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.018056 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-token" pod "cfcceda1-486c-48fe-a25e-d7d37f97e5a3" (UID: "cfcceda1-486c-48fe-a25e-d7d37f97e5a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:39.018563 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.018534 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-envoy" pod "cfcceda1-486c-48fe-a25e-d7d37f97e5a3" (UID: "cfcceda1-486c-48fe-a25e-d7d37f97e5a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:39.019742 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.019684 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-podinfo" pod "cfcceda1-486c-48fe-a25e-d7d37f97e5a3" (UID: "cfcceda1-486c-48fe-a25e-d7d37f97e5a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:25:39.119755 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.118475 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqfmd\" (UniqueName: \"kubernetes.io/projected/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-kube-api-access-nqfmd\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:39.119755 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.118510 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:39.119755 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.118524 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:39.119755 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.118560 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:39.119755 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.118574 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:39.119755 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.118588 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:39.119755 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.118600 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:39.119755 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.118613 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfcceda1-486c-48fe-a25e-d7d37f97e5a3-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:39.592805 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.592769 4947 generic.go:356] "Generic (PLEG): container finished" podID="cfcceda1-486c-48fe-a25e-d7d37f97e5a3" containerID="d6e8d70153e827d4e5eeb57bf67a7272f2a0ef232bcb615d65c6c5e37c9ac3d8" exitCode=0 Nov 03 17:25:39.592972 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.592819 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" Nov 03 17:25:39.592972 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.592858 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" event={"ID":"cfcceda1-486c-48fe-a25e-d7d37f97e5a3","Type":"ContainerDied","Data":"d6e8d70153e827d4e5eeb57bf67a7272f2a0ef232bcb615d65c6c5e37c9ac3d8"} Nov 03 17:25:39.592972 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.592921 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs" event={"ID":"cfcceda1-486c-48fe-a25e-d7d37f97e5a3","Type":"ContainerDied","Data":"2506d29dcd12280f2f791b6719402e5d66d65700e4b27203373f49b7b1ef4988"} Nov 03 17:25:39.592972 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.592956 4947 scope.go:122] "RemoveContainer" containerID="d6e8d70153e827d4e5eeb57bf67a7272f2a0ef232bcb615d65c6c5e37c9ac3d8" Nov 03 17:25:39.605553 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.605535 4947 scope.go:122] "RemoveContainer" containerID="d6e8d70153e827d4e5eeb57bf67a7272f2a0ef232bcb615d65c6c5e37c9ac3d8" Nov 03 17:25:39.605899 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:25:39.605878 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6e8d70153e827d4e5eeb57bf67a7272f2a0ef232bcb615d65c6c5e37c9ac3d8\": container with ID starting with d6e8d70153e827d4e5eeb57bf67a7272f2a0ef232bcb615d65c6c5e37c9ac3d8 not found: ID does not exist" containerID="d6e8d70153e827d4e5eeb57bf67a7272f2a0ef232bcb615d65c6c5e37c9ac3d8" Nov 03 17:25:39.605986 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.605910 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6e8d70153e827d4e5eeb57bf67a7272f2a0ef232bcb615d65c6c5e37c9ac3d8"} err="failed to get container status \"d6e8d70153e827d4e5eeb57bf67a7272f2a0ef232bcb615d65c6c5e37c9ac3d8\": rpc error: code = NotFound desc = could not find container \"d6e8d70153e827d4e5eeb57bf67a7272f2a0ef232bcb615d65c6c5e37c9ac3d8\": container with ID starting with d6e8d70153e827d4e5eeb57bf67a7272f2a0ef232bcb615d65c6c5e37c9ac3d8 not found: ID does not exist" Nov 03 17:25:39.649529 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.649500 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs"] Nov 03 17:25:39.662595 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:39.662563 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-r8jhs"] Nov 03 17:25:40.489144 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:40.489108 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:25:40.566924 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:40.566895 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfcceda1-486c-48fe-a25e-d7d37f97e5a3" path="/var/lib/kubelet/pods/cfcceda1-486c-48fe-a25e-d7d37f97e5a3/volumes" Nov 03 17:25:44.920077 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:44.920033 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" podUID="6318894b-7e77-47fa-805e-a28d53c1fead" containerName="main" containerID="cri-o://e36b4a77090bc5a35d346fb47bbe97104c501673790f36b78368080b06eb3c87" gracePeriod=45 Nov 03 17:25:45.200362 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.200332 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:25:45.238330 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.238298 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-model-cache\") pod \"6318894b-7e77-47fa-805e-a28d53c1fead\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " Nov 03 17:25:45.238462 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.238344 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-tmp-dir\") pod \"6318894b-7e77-47fa-805e-a28d53c1fead\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " Nov 03 17:25:45.238462 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.238408 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6318894b-7e77-47fa-805e-a28d53c1fead-tls-certs\" (UniqueName: \"kubernetes.io/secret/6318894b-7e77-47fa-805e-a28d53c1fead-tls-certs\") pod \"6318894b-7e77-47fa-805e-a28d53c1fead\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " Nov 03 17:25:45.238462 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.238447 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6318894b-7e77-47fa-805e-a28d53c1fead-kube-api-access-pvvdx\" (UniqueName: \"kubernetes.io/projected/6318894b-7e77-47fa-805e-a28d53c1fead-kube-api-access-pvvdx\") pod \"6318894b-7e77-47fa-805e-a28d53c1fead\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " Nov 03 17:25:45.238573 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.238473 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-home\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-home\") pod \"6318894b-7e77-47fa-805e-a28d53c1fead\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " Nov 03 17:25:45.238625 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.238577 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-dshm\") pod \"6318894b-7e77-47fa-805e-a28d53c1fead\" (UID: \"6318894b-7e77-47fa-805e-a28d53c1fead\") " Nov 03 17:25:45.238685 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.238570 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-model-cache" pod "6318894b-7e77-47fa-805e-a28d53c1fead" (UID: "6318894b-7e77-47fa-805e-a28d53c1fead"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:45.238793 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.238618 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-tmp-dir" pod "6318894b-7e77-47fa-805e-a28d53c1fead" (UID: "6318894b-7e77-47fa-805e-a28d53c1fead"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:45.238861 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.238819 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-home" pod "6318894b-7e77-47fa-805e-a28d53c1fead" (UID: "6318894b-7e77-47fa-805e-a28d53c1fead"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:45.239334 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.239311 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:45.239661 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.239638 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:45.239661 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.239661 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:45.241377 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.241312 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6318894b-7e77-47fa-805e-a28d53c1fead-tls-certs" pod "6318894b-7e77-47fa-805e-a28d53c1fead" (UID: "6318894b-7e77-47fa-805e-a28d53c1fead"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:25:45.241486 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.241407 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-dshm" pod "6318894b-7e77-47fa-805e-a28d53c1fead" (UID: "6318894b-7e77-47fa-805e-a28d53c1fead"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:45.241612 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.241587 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6318894b-7e77-47fa-805e-a28d53c1fead-kube-api-access-pvvdx" pod "6318894b-7e77-47fa-805e-a28d53c1fead" (UID: "6318894b-7e77-47fa-805e-a28d53c1fead"). InnerVolumeSpecName "kube-api-access-pvvdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:45.341325 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.341298 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6318894b-7e77-47fa-805e-a28d53c1fead-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:45.341325 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.341324 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6318894b-7e77-47fa-805e-a28d53c1fead-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:45.341325 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.341335 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvvdx\" (UniqueName: \"kubernetes.io/projected/6318894b-7e77-47fa-805e-a28d53c1fead-kube-api-access-pvvdx\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:45.655983 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.655948 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct"] Nov 03 17:25:45.656240 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.656214 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" podUID="c7a45866-e8e8-4731-b688-4674bf874a44" containerName="istio-proxy" containerID="cri-o://6da70a148d77c7926305be582fc27a381aee3d79064ef17403d98f1fcbc39f2f" gracePeriod=30 Nov 03 17:25:45.658107 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.658060 4947 generic.go:356] "Generic (PLEG): container finished" podID="6318894b-7e77-47fa-805e-a28d53c1fead" containerID="e36b4a77090bc5a35d346fb47bbe97104c501673790f36b78368080b06eb3c87" exitCode=0 Nov 03 17:25:45.658236 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.658119 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" Nov 03 17:25:45.658418 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.658118 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" event={"ID":"6318894b-7e77-47fa-805e-a28d53c1fead","Type":"ContainerDied","Data":"e36b4a77090bc5a35d346fb47bbe97104c501673790f36b78368080b06eb3c87"} Nov 03 17:25:45.658589 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.658540 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2" event={"ID":"6318894b-7e77-47fa-805e-a28d53c1fead","Type":"ContainerDied","Data":"d2ca02bbfddc8888af346474fff22f3e913696cf5cc15b66df13188b0f2b8ae4"} Nov 03 17:25:45.658589 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.658579 4947 scope.go:122] "RemoveContainer" containerID="e36b4a77090bc5a35d346fb47bbe97104c501673790f36b78368080b06eb3c87" Nov 03 17:25:45.672370 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.672354 4947 scope.go:122] "RemoveContainer" containerID="e36b4a77090bc5a35d346fb47bbe97104c501673790f36b78368080b06eb3c87" Nov 03 17:25:45.672845 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:25:45.672678 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36b4a77090bc5a35d346fb47bbe97104c501673790f36b78368080b06eb3c87\": container with ID starting with e36b4a77090bc5a35d346fb47bbe97104c501673790f36b78368080b06eb3c87 not found: ID does not exist" containerID="e36b4a77090bc5a35d346fb47bbe97104c501673790f36b78368080b06eb3c87" Nov 03 17:25:45.672845 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.672728 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36b4a77090bc5a35d346fb47bbe97104c501673790f36b78368080b06eb3c87"} err="failed to get container status \"e36b4a77090bc5a35d346fb47bbe97104c501673790f36b78368080b06eb3c87\": rpc error: code = NotFound desc = could not find container \"e36b4a77090bc5a35d346fb47bbe97104c501673790f36b78368080b06eb3c87\": container with ID starting with e36b4a77090bc5a35d346fb47bbe97104c501673790f36b78368080b06eb3c87 not found: ID does not exist" Nov 03 17:25:45.814727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.809187 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2"] Nov 03 17:25:45.814727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:45.810981 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-547925/test-model-547925-kserve-77c46fdc5-q9fz2"] Nov 03 17:25:46.567063 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:46.567027 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6318894b-7e77-47fa-805e-a28d53c1fead" path="/var/lib/kubelet/pods/6318894b-7e77-47fa-805e-a28d53c1fead/volumes" Nov 03 17:25:47.956092 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:47.956046 4947 patch_prober.go:29] interesting pod/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:25:47.956504 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:47.956106 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" podUID="c7a45866-e8e8-4731-b688-4674bf874a44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:25:51.003245 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.003221 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:25:51.070383 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.070347 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7a45866-e8e8-4731-b688-4674bf874a44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a45866-e8e8-4731-b688-4674bf874a44-istio-podinfo\") pod \"c7a45866-e8e8-4731-b688-4674bf874a44\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " Nov 03 17:25:51.070383 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.070387 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-istio-data\") pod \"c7a45866-e8e8-4731-b688-4674bf874a44\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " Nov 03 17:25:51.070608 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.070439 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-workload-socket\") pod \"c7a45866-e8e8-4731-b688-4674bf874a44\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " Nov 03 17:25:51.070608 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.070462 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-credential-socket\") pod \"c7a45866-e8e8-4731-b688-4674bf874a44\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " Nov 03 17:25:51.070608 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.070585 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a45866-e8e8-4731-b688-4674bf874a44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a45866-e8e8-4731-b688-4674bf874a44-istiod-ca-cert\") pod \"c7a45866-e8e8-4731-b688-4674bf874a44\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " Nov 03 17:25:51.070793 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.070623 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-workload-certs\") pod \"c7a45866-e8e8-4731-b688-4674bf874a44\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " Nov 03 17:25:51.070793 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.070674 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a45866-e8e8-4731-b688-4674bf874a44-istio-token\" (UniqueName: \"kubernetes.io/projected/c7a45866-e8e8-4731-b688-4674bf874a44-istio-token\") pod \"c7a45866-e8e8-4731-b688-4674bf874a44\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " Nov 03 17:25:51.070793 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.070770 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-istio-data" pod "c7a45866-e8e8-4731-b688-4674bf874a44" (UID: "c7a45866-e8e8-4731-b688-4674bf874a44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:51.070793 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.070782 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-istio-envoy\") pod \"c7a45866-e8e8-4731-b688-4674bf874a44\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " Nov 03 17:25:51.071017 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.070787 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-workload-socket" pod "c7a45866-e8e8-4731-b688-4674bf874a44" (UID: "c7a45866-e8e8-4731-b688-4674bf874a44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:51.071017 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.070817 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-credential-socket" pod "c7a45866-e8e8-4731-b688-4674bf874a44" (UID: "c7a45866-e8e8-4731-b688-4674bf874a44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:51.071017 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.070851 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a45866-e8e8-4731-b688-4674bf874a44-kube-api-access-wms95\" (UniqueName: \"kubernetes.io/projected/c7a45866-e8e8-4731-b688-4674bf874a44-kube-api-access-wms95\") pod \"c7a45866-e8e8-4731-b688-4674bf874a44\" (UID: \"c7a45866-e8e8-4731-b688-4674bf874a44\") " Nov 03 17:25:51.071017 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.070976 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-workload-certs" pod "c7a45866-e8e8-4731-b688-4674bf874a44" (UID: "c7a45866-e8e8-4731-b688-4674bf874a44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:51.071661 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.071360 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a45866-e8e8-4731-b688-4674bf874a44-istiod-ca-cert" pod "c7a45866-e8e8-4731-b688-4674bf874a44" (UID: "c7a45866-e8e8-4731-b688-4674bf874a44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:25:51.071661 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.071582 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:51.071661 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.071603 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:51.071661 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.071616 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:51.071661 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.071629 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a45866-e8e8-4731-b688-4674bf874a44-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:51.071661 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.071642 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:51.073088 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.073063 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7a45866-e8e8-4731-b688-4674bf874a44-istio-podinfo" pod "c7a45866-e8e8-4731-b688-4674bf874a44" (UID: "c7a45866-e8e8-4731-b688-4674bf874a44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:25:51.073236 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.073217 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-istio-envoy" pod "c7a45866-e8e8-4731-b688-4674bf874a44" (UID: "c7a45866-e8e8-4731-b688-4674bf874a44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:25:51.073285 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.073251 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a45866-e8e8-4731-b688-4674bf874a44-kube-api-access-wms95" pod "c7a45866-e8e8-4731-b688-4674bf874a44" (UID: "c7a45866-e8e8-4731-b688-4674bf874a44"). InnerVolumeSpecName "kube-api-access-wms95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:51.073412 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.073398 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a45866-e8e8-4731-b688-4674bf874a44-istio-token" pod "c7a45866-e8e8-4731-b688-4674bf874a44" (UID: "c7a45866-e8e8-4731-b688-4674bf874a44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:25:51.172981 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.172946 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a45866-e8e8-4731-b688-4674bf874a44-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:51.172981 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.172977 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a45866-e8e8-4731-b688-4674bf874a44-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:51.172981 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.172990 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wms95\" (UniqueName: \"kubernetes.io/projected/c7a45866-e8e8-4731-b688-4674bf874a44-kube-api-access-wms95\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:51.173249 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.173000 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a45866-e8e8-4731-b688-4674bf874a44-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:25:51.713423 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.713393 4947 generic.go:356] "Generic (PLEG): container finished" podID="c7a45866-e8e8-4731-b688-4674bf874a44" containerID="6da70a148d77c7926305be582fc27a381aee3d79064ef17403d98f1fcbc39f2f" exitCode=0 Nov 03 17:25:51.713622 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.713479 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" Nov 03 17:25:51.713622 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.713486 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" event={"ID":"c7a45866-e8e8-4731-b688-4674bf874a44","Type":"ContainerDied","Data":"6da70a148d77c7926305be582fc27a381aee3d79064ef17403d98f1fcbc39f2f"} Nov 03 17:25:51.713622 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.713549 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct" event={"ID":"c7a45866-e8e8-4731-b688-4674bf874a44","Type":"ContainerDied","Data":"4ff6884ee1a6473b7bfeca7bb3df269dd7d359f406647c125e92473abed28015"} Nov 03 17:25:51.713622 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.713589 4947 scope.go:122] "RemoveContainer" containerID="6da70a148d77c7926305be582fc27a381aee3d79064ef17403d98f1fcbc39f2f" Nov 03 17:25:51.725410 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.725393 4947 scope.go:122] "RemoveContainer" containerID="60ef2342f8d84e076a0b8bffa2ac96049cdd911af75d4f35580ee3e100d003aa" Nov 03 17:25:51.735638 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.735620 4947 scope.go:122] "RemoveContainer" containerID="6da70a148d77c7926305be582fc27a381aee3d79064ef17403d98f1fcbc39f2f" Nov 03 17:25:51.735963 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:25:51.735945 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da70a148d77c7926305be582fc27a381aee3d79064ef17403d98f1fcbc39f2f\": container with ID starting with 6da70a148d77c7926305be582fc27a381aee3d79064ef17403d98f1fcbc39f2f not found: ID does not exist" containerID="6da70a148d77c7926305be582fc27a381aee3d79064ef17403d98f1fcbc39f2f" Nov 03 17:25:51.736035 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.735968 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da70a148d77c7926305be582fc27a381aee3d79064ef17403d98f1fcbc39f2f"} err="failed to get container status \"6da70a148d77c7926305be582fc27a381aee3d79064ef17403d98f1fcbc39f2f\": rpc error: code = NotFound desc = could not find container \"6da70a148d77c7926305be582fc27a381aee3d79064ef17403d98f1fcbc39f2f\": container with ID starting with 6da70a148d77c7926305be582fc27a381aee3d79064ef17403d98f1fcbc39f2f not found: ID does not exist" Nov 03 17:25:51.736035 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.735982 4947 scope.go:122] "RemoveContainer" containerID="60ef2342f8d84e076a0b8bffa2ac96049cdd911af75d4f35580ee3e100d003aa" Nov 03 17:25:51.736249 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:25:51.736229 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60ef2342f8d84e076a0b8bffa2ac96049cdd911af75d4f35580ee3e100d003aa\": container with ID starting with 60ef2342f8d84e076a0b8bffa2ac96049cdd911af75d4f35580ee3e100d003aa not found: ID does not exist" containerID="60ef2342f8d84e076a0b8bffa2ac96049cdd911af75d4f35580ee3e100d003aa" Nov 03 17:25:51.736291 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.736256 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ef2342f8d84e076a0b8bffa2ac96049cdd911af75d4f35580ee3e100d003aa"} err="failed to get container status \"60ef2342f8d84e076a0b8bffa2ac96049cdd911af75d4f35580ee3e100d003aa\": rpc error: code = NotFound desc = could not find container \"60ef2342f8d84e076a0b8bffa2ac96049cdd911af75d4f35580ee3e100d003aa\": container with ID starting with 60ef2342f8d84e076a0b8bffa2ac96049cdd911af75d4f35580ee3e100d003aa not found: ID does not exist" Nov 03 17:25:51.787433 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.787394 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct"] Nov 03 17:25:51.790971 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:51.790944 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-cb0fa3-openshift-default-86dc4b5bf-zltct"] Nov 03 17:25:52.367755 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:52.367724 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9"] Nov 03 17:25:52.568210 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:52.568182 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a45866-e8e8-4731-b688-4674bf874a44" path="/var/lib/kubelet/pods/c7a45866-e8e8-4731-b688-4674bf874a44/volumes" Nov 03 17:25:57.368876 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:25:57.368821 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" podUID="407a82a9-162d-4120-82a1-f6022279dced" containerName="payload-pre-processing" containerID="cri-o://a82226526df3d3f91339630ab15d1253141d3f36c17b4a5d62d40855416c9c0d" gracePeriod=25 Nov 03 17:26:00.130325 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:26:00.130283 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:26:07.349752 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:07.349251 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" podUID="407a82a9-162d-4120-82a1-f6022279dced" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:26:08.572388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:08.572354 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz"] Nov 03 17:26:08.573010 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:08.572598 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" podUID="1e32622d-7ba0-45ad-a866-05089e15f14e" containerName="maas-api" containerID="cri-o://b9b3e12c43d7301555a8cec805a39b21faba309a861c175fd462b77b82048cd0" gracePeriod=30 Nov 03 17:26:08.615096 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:08.615064 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" podUID="1e32622d-7ba0-45ad-a866-05089e15f14e" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.1.5:8443/health\": dial tcp 10.132.1.5:8443: connect: connection refused" Nov 03 17:26:08.892164 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:08.892118 4947 generic.go:356] "Generic (PLEG): container finished" podID="1e32622d-7ba0-45ad-a866-05089e15f14e" containerID="b9b3e12c43d7301555a8cec805a39b21faba309a861c175fd462b77b82048cd0" exitCode=0 Nov 03 17:26:08.892612 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:08.892350 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" event={"ID":"1e32622d-7ba0-45ad-a866-05089e15f14e","Type":"ContainerDied","Data":"b9b3e12c43d7301555a8cec805a39b21faba309a861c175fd462b77b82048cd0"} Nov 03 17:26:08.922953 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:08.922922 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" Nov 03 17:26:09.083781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.080388 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e32622d-7ba0-45ad-a866-05089e15f14e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e32622d-7ba0-45ad-a866-05089e15f14e-maas-api-tls\") pod \"1e32622d-7ba0-45ad-a866-05089e15f14e\" (UID: \"1e32622d-7ba0-45ad-a866-05089e15f14e\") " Nov 03 17:26:09.083781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.080536 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e32622d-7ba0-45ad-a866-05089e15f14e-kube-api-access-6ttkh\" (UniqueName: \"kubernetes.io/projected/1e32622d-7ba0-45ad-a866-05089e15f14e-kube-api-access-6ttkh\") pod \"1e32622d-7ba0-45ad-a866-05089e15f14e\" (UID: \"1e32622d-7ba0-45ad-a866-05089e15f14e\") " Nov 03 17:26:09.083781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.080680 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e32622d-7ba0-45ad-a866-05089e15f14e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e32622d-7ba0-45ad-a866-05089e15f14e-metrics-certs\") pod \"1e32622d-7ba0-45ad-a866-05089e15f14e\" (UID: \"1e32622d-7ba0-45ad-a866-05089e15f14e\") " Nov 03 17:26:09.083781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.080788 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e32622d-7ba0-45ad-a866-05089e15f14e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e32622d-7ba0-45ad-a866-05089e15f14e-openshift-service-ca\") pod \"1e32622d-7ba0-45ad-a866-05089e15f14e\" (UID: \"1e32622d-7ba0-45ad-a866-05089e15f14e\") " Nov 03 17:26:09.104071 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.103523 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e32622d-7ba0-45ad-a866-05089e15f14e-openshift-service-ca" pod "1e32622d-7ba0-45ad-a866-05089e15f14e" (UID: "1e32622d-7ba0-45ad-a866-05089e15f14e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:26:09.116959 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.108962 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4"] Nov 03 17:26:09.116959 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.109191 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" podUID="23ddeb2f-f94f-4012-a987-259a185e19aa" containerName="maas-api" containerID="cri-o://fcd897d7d58db82d16cec1dc1eabfa815893cf225bd9aad16c05a46e2509ece8" gracePeriod=30 Nov 03 17:26:09.128054 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.128017 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e32622d-7ba0-45ad-a866-05089e15f14e-metrics-certs" pod "1e32622d-7ba0-45ad-a866-05089e15f14e" (UID: "1e32622d-7ba0-45ad-a866-05089e15f14e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:26:09.136898 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.131889 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e32622d-7ba0-45ad-a866-05089e15f14e-maas-api-tls" pod "1e32622d-7ba0-45ad-a866-05089e15f14e" (UID: "1e32622d-7ba0-45ad-a866-05089e15f14e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:26:09.141035 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.141006 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e32622d-7ba0-45ad-a866-05089e15f14e-kube-api-access-6ttkh" pod "1e32622d-7ba0-45ad-a866-05089e15f14e" (UID: "1e32622d-7ba0-45ad-a866-05089e15f14e"). InnerVolumeSpecName "kube-api-access-6ttkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:09.183304 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.183275 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ttkh\" (UniqueName: \"kubernetes.io/projected/1e32622d-7ba0-45ad-a866-05089e15f14e-kube-api-access-6ttkh\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:09.183304 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.183305 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e32622d-7ba0-45ad-a866-05089e15f14e-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:09.183525 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.183315 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e32622d-7ba0-45ad-a866-05089e15f14e-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:09.183525 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.183323 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e32622d-7ba0-45ad-a866-05089e15f14e-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:09.573818 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.573787 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:26:09.701945 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.701916 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23ddeb2f-f94f-4012-a987-259a185e19aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ddeb2f-f94f-4012-a987-259a185e19aa-metrics-certs\") pod \"23ddeb2f-f94f-4012-a987-259a185e19aa\" (UID: \"23ddeb2f-f94f-4012-a987-259a185e19aa\") " Nov 03 17:26:09.702135 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.702045 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23ddeb2f-f94f-4012-a987-259a185e19aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ddeb2f-f94f-4012-a987-259a185e19aa-maas-api-tls\") pod \"23ddeb2f-f94f-4012-a987-259a185e19aa\" (UID: \"23ddeb2f-f94f-4012-a987-259a185e19aa\") " Nov 03 17:26:09.702135 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.702082 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ddeb2f-f94f-4012-a987-259a185e19aa-kube-api-access-rfvnc\" (UniqueName: \"kubernetes.io/projected/23ddeb2f-f94f-4012-a987-259a185e19aa-kube-api-access-rfvnc\") pod \"23ddeb2f-f94f-4012-a987-259a185e19aa\" (UID: \"23ddeb2f-f94f-4012-a987-259a185e19aa\") " Nov 03 17:26:09.702135 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.702109 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23ddeb2f-f94f-4012-a987-259a185e19aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ddeb2f-f94f-4012-a987-259a185e19aa-openshift-service-ca\") pod \"23ddeb2f-f94f-4012-a987-259a185e19aa\" (UID: \"23ddeb2f-f94f-4012-a987-259a185e19aa\") " Nov 03 17:26:09.702639 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.702564 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ddeb2f-f94f-4012-a987-259a185e19aa-openshift-service-ca" pod "23ddeb2f-f94f-4012-a987-259a185e19aa" (UID: "23ddeb2f-f94f-4012-a987-259a185e19aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:26:09.702934 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.702871 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ddeb2f-f94f-4012-a987-259a185e19aa-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:09.704243 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.704217 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ddeb2f-f94f-4012-a987-259a185e19aa-kube-api-access-rfvnc" pod "23ddeb2f-f94f-4012-a987-259a185e19aa" (UID: "23ddeb2f-f94f-4012-a987-259a185e19aa"). InnerVolumeSpecName "kube-api-access-rfvnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:09.704354 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.704253 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ddeb2f-f94f-4012-a987-259a185e19aa-metrics-certs" pod "23ddeb2f-f94f-4012-a987-259a185e19aa" (UID: "23ddeb2f-f94f-4012-a987-259a185e19aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:26:09.704354 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.704263 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23ddeb2f-f94f-4012-a987-259a185e19aa-maas-api-tls" pod "23ddeb2f-f94f-4012-a987-259a185e19aa" (UID: "23ddeb2f-f94f-4012-a987-259a185e19aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:26:09.804301 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.804268 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23ddeb2f-f94f-4012-a987-259a185e19aa-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:09.804301 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.804297 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23ddeb2f-f94f-4012-a987-259a185e19aa-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:09.804301 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.804307 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfvnc\" (UniqueName: \"kubernetes.io/projected/23ddeb2f-f94f-4012-a987-259a185e19aa-kube-api-access-rfvnc\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:09.945781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.931626 4947 generic.go:356] "Generic (PLEG): container finished" podID="23ddeb2f-f94f-4012-a987-259a185e19aa" containerID="fcd897d7d58db82d16cec1dc1eabfa815893cf225bd9aad16c05a46e2509ece8" exitCode=0 Nov 03 17:26:09.945781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.931875 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" event={"ID":"23ddeb2f-f94f-4012-a987-259a185e19aa","Type":"ContainerDied","Data":"fcd897d7d58db82d16cec1dc1eabfa815893cf225bd9aad16c05a46e2509ece8"} Nov 03 17:26:09.945781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.931935 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" event={"ID":"23ddeb2f-f94f-4012-a987-259a185e19aa","Type":"ContainerDied","Data":"44084e7ebd711d3e6e19c2a0e90b550ff88200698e8f037f099f7e93239fc4d8"} Nov 03 17:26:09.945781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.931965 4947 scope.go:122] "RemoveContainer" containerID="fcd897d7d58db82d16cec1dc1eabfa815893cf225bd9aad16c05a46e2509ece8" Nov 03 17:26:09.945781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.932135 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4" Nov 03 17:26:09.945781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.942969 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" event={"ID":"1e32622d-7ba0-45ad-a866-05089e15f14e","Type":"ContainerDied","Data":"a99ad3b761b78191e6ce4ba56ab06322d275519d5204cbb569ef0fde326b0f59"} Nov 03 17:26:09.945781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.943285 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz" Nov 03 17:26:09.966024 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.965972 4947 scope.go:122] "RemoveContainer" containerID="fcd897d7d58db82d16cec1dc1eabfa815893cf225bd9aad16c05a46e2509ece8" Nov 03 17:26:09.966812 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:26:09.966435 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd897d7d58db82d16cec1dc1eabfa815893cf225bd9aad16c05a46e2509ece8\": container with ID starting with fcd897d7d58db82d16cec1dc1eabfa815893cf225bd9aad16c05a46e2509ece8 not found: ID does not exist" containerID="fcd897d7d58db82d16cec1dc1eabfa815893cf225bd9aad16c05a46e2509ece8" Nov 03 17:26:09.966812 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.966484 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd897d7d58db82d16cec1dc1eabfa815893cf225bd9aad16c05a46e2509ece8"} err="failed to get container status \"fcd897d7d58db82d16cec1dc1eabfa815893cf225bd9aad16c05a46e2509ece8\": rpc error: code = NotFound desc = could not find container \"fcd897d7d58db82d16cec1dc1eabfa815893cf225bd9aad16c05a46e2509ece8\": container with ID starting with fcd897d7d58db82d16cec1dc1eabfa815893cf225bd9aad16c05a46e2509ece8 not found: ID does not exist" Nov 03 17:26:09.966812 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:09.966509 4947 scope.go:122] "RemoveContainer" containerID="b9b3e12c43d7301555a8cec805a39b21faba309a861c175fd462b77b82048cd0" Nov 03 17:26:10.016894 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:10.016868 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4"] Nov 03 17:26:10.023848 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:10.023824 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-44736d-767cc9bf4-kglh4"] Nov 03 17:26:10.059214 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:10.059183 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz"] Nov 03 17:26:10.063061 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:10.063039 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-527051-7d47c45df9-rlvmz"] Nov 03 17:26:10.566328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:10.566288 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e32622d-7ba0-45ad-a866-05089e15f14e" path="/var/lib/kubelet/pods/1e32622d-7ba0-45ad-a866-05089e15f14e/volumes" Nov 03 17:26:10.566823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:10.566801 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23ddeb2f-f94f-4012-a987-259a185e19aa" path="/var/lib/kubelet/pods/23ddeb2f-f94f-4012-a987-259a185e19aa/volumes" Nov 03 17:26:17.345247 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:17.345210 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" podUID="407a82a9-162d-4120-82a1-f6022279dced" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:26:17.654624 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:17.654601 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" Nov 03 17:26:17.779162 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:17.779132 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/407a82a9-162d-4120-82a1-f6022279dced-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407a82a9-162d-4120-82a1-f6022279dced-plugins-config-volume\") pod \"407a82a9-162d-4120-82a1-f6022279dced\" (UID: \"407a82a9-162d-4120-82a1-f6022279dced\") " Nov 03 17:26:17.779374 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:17.779291 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/407a82a9-162d-4120-82a1-f6022279dced-kube-api-access-c65hk\" (UniqueName: \"kubernetes.io/projected/407a82a9-162d-4120-82a1-f6022279dced-kube-api-access-c65hk\") pod \"407a82a9-162d-4120-82a1-f6022279dced\" (UID: \"407a82a9-162d-4120-82a1-f6022279dced\") " Nov 03 17:26:17.779511 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:17.779487 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/407a82a9-162d-4120-82a1-f6022279dced-plugins-config-volume" pod "407a82a9-162d-4120-82a1-f6022279dced" (UID: "407a82a9-162d-4120-82a1-f6022279dced"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:26:17.779888 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:17.779867 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/407a82a9-162d-4120-82a1-f6022279dced-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:17.781664 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:17.781633 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/407a82a9-162d-4120-82a1-f6022279dced-kube-api-access-c65hk" pod "407a82a9-162d-4120-82a1-f6022279dced" (UID: "407a82a9-162d-4120-82a1-f6022279dced"). InnerVolumeSpecName "kube-api-access-c65hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:17.881532 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:17.881455 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c65hk\" (UniqueName: \"kubernetes.io/projected/407a82a9-162d-4120-82a1-f6022279dced-kube-api-access-c65hk\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:18.046292 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:18.046252 4947 generic.go:356] "Generic (PLEG): container finished" podID="407a82a9-162d-4120-82a1-f6022279dced" containerID="a82226526df3d3f91339630ab15d1253141d3f36c17b4a5d62d40855416c9c0d" exitCode=0 Nov 03 17:26:18.046459 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:18.046352 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" Nov 03 17:26:18.046459 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:18.046387 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" event={"ID":"407a82a9-162d-4120-82a1-f6022279dced","Type":"ContainerDied","Data":"a82226526df3d3f91339630ab15d1253141d3f36c17b4a5d62d40855416c9c0d"} Nov 03 17:26:18.046459 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:18.046442 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9" event={"ID":"407a82a9-162d-4120-82a1-f6022279dced","Type":"ContainerDied","Data":"06a1a1238f7c23048a604aab1649927b9b15c744562ed9baebc7ef24fd9be360"} Nov 03 17:26:18.046581 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:18.046465 4947 scope.go:122] "RemoveContainer" containerID="a82226526df3d3f91339630ab15d1253141d3f36c17b4a5d62d40855416c9c0d" Nov 03 17:26:18.058374 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:18.058357 4947 scope.go:122] "RemoveContainer" containerID="a82226526df3d3f91339630ab15d1253141d3f36c17b4a5d62d40855416c9c0d" Nov 03 17:26:18.058662 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:26:18.058633 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a82226526df3d3f91339630ab15d1253141d3f36c17b4a5d62d40855416c9c0d\": container with ID starting with a82226526df3d3f91339630ab15d1253141d3f36c17b4a5d62d40855416c9c0d not found: ID does not exist" containerID="a82226526df3d3f91339630ab15d1253141d3f36c17b4a5d62d40855416c9c0d" Nov 03 17:26:18.058818 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:18.058668 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82226526df3d3f91339630ab15d1253141d3f36c17b4a5d62d40855416c9c0d"} err="failed to get container status \"a82226526df3d3f91339630ab15d1253141d3f36c17b4a5d62d40855416c9c0d\": rpc error: code = NotFound desc = could not find container \"a82226526df3d3f91339630ab15d1253141d3f36c17b4a5d62d40855416c9c0d\": container with ID starting with a82226526df3d3f91339630ab15d1253141d3f36c17b4a5d62d40855416c9c0d not found: ID does not exist" Nov 03 17:26:18.108363 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:18.108338 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9"] Nov 03 17:26:18.119283 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:18.119257 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-547925-8484d89d9f-xq2h9"] Nov 03 17:26:18.567099 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:18.567071 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="407a82a9-162d-4120-82a1-f6022279dced" path="/var/lib/kubelet/pods/407a82a9-162d-4120-82a1-f6022279dced/volumes" Nov 03 17:26:19.116409 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:19.116372 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn"] Nov 03 17:26:19.117828 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:19.117791 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" podUID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" containerName="istio-proxy" containerID="cri-o://33028b9510697cbe5c77d92f8f584ffcab23ec66c60730bf4fc12f7c3c6a864a" gracePeriod=30 Nov 03 17:26:22.464018 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:22.463978 4947 patch_prober.go:29] interesting pod/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:26:22.464438 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:22.464044 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" podUID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:26:24.412641 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.412613 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:26:24.483960 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.483914 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-workload-socket\") pod \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " Nov 03 17:26:24.484158 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.483982 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-token\" (UniqueName: \"kubernetes.io/projected/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-token\") pod \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " Nov 03 17:26:24.484158 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.484031 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-workload-certs\") pod \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " Nov 03 17:26:24.484158 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.484070 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-podinfo\") pod \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " Nov 03 17:26:24.484158 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.484124 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-kube-api-access-t47xx\" (UniqueName: \"kubernetes.io/projected/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-kube-api-access-t47xx\") pod \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " Nov 03 17:26:24.484398 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.484170 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-data\") pod \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " Nov 03 17:26:24.484398 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.484209 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istiod-ca-cert\") pod \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " Nov 03 17:26:24.484398 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.484205 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-workload-socket" pod "048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" (UID: "048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:24.484569 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.484396 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-workload-certs" pod "048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" (UID: "048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:24.484569 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.484416 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-credential-socket\") pod \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " Nov 03 17:26:24.484569 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.484474 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-envoy\") pod \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\" (UID: \"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d\") " Nov 03 17:26:24.484790 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.484645 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-credential-socket" pod "048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" (UID: "048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:24.484862 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.484846 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-data" pod "048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" (UID: "048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:24.484921 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.484751 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istiod-ca-cert" pod "048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" (UID: "048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:26:24.485328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.485254 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:24.485328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.485285 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:24.485328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.485299 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:24.485328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.485314 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:24.485328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.485326 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:24.486842 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.486816 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-token" pod "048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" (UID: "048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:24.486950 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.486887 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-envoy" pod "048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" (UID: "048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:24.487351 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.487328 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-kube-api-access-t47xx" pod "048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" (UID: "048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d"). InnerVolumeSpecName "kube-api-access-t47xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:24.487432 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.487363 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-podinfo" pod "048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" (UID: "048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:26:24.586381 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.586353 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:24.586381 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.586381 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:24.586541 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.586390 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:24.586541 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:24.586398 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t47xx\" (UniqueName: \"kubernetes.io/projected/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d-kube-api-access-t47xx\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:25.097989 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:25.097947 4947 generic.go:356] "Generic (PLEG): container finished" podID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" containerID="33028b9510697cbe5c77d92f8f584ffcab23ec66c60730bf4fc12f7c3c6a864a" exitCode=0 Nov 03 17:26:25.098219 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:25.098026 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" Nov 03 17:26:25.098219 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:25.098052 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" event={"ID":"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d","Type":"ContainerDied","Data":"33028b9510697cbe5c77d92f8f584ffcab23ec66c60730bf4fc12f7c3c6a864a"} Nov 03 17:26:25.098219 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:25.098179 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn" event={"ID":"048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d","Type":"ContainerDied","Data":"6de2c686380ab14684ad7d06ea44f18b1d07235e4ada164c5dd34d5f4f7348fe"} Nov 03 17:26:25.098219 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:25.098205 4947 scope.go:122] "RemoveContainer" containerID="33028b9510697cbe5c77d92f8f584ffcab23ec66c60730bf4fc12f7c3c6a864a" Nov 03 17:26:25.109239 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:25.109219 4947 scope.go:122] "RemoveContainer" containerID="fda76cd77932b7436280edfeff09a7edebabc200d13dfb1bbcc274088f8be176" Nov 03 17:26:25.118582 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:25.118562 4947 scope.go:122] "RemoveContainer" containerID="33028b9510697cbe5c77d92f8f584ffcab23ec66c60730bf4fc12f7c3c6a864a" Nov 03 17:26:25.118965 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:26:25.118940 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33028b9510697cbe5c77d92f8f584ffcab23ec66c60730bf4fc12f7c3c6a864a\": container with ID starting with 33028b9510697cbe5c77d92f8f584ffcab23ec66c60730bf4fc12f7c3c6a864a not found: ID does not exist" containerID="33028b9510697cbe5c77d92f8f584ffcab23ec66c60730bf4fc12f7c3c6a864a" Nov 03 17:26:25.119065 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:25.118971 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33028b9510697cbe5c77d92f8f584ffcab23ec66c60730bf4fc12f7c3c6a864a"} err="failed to get container status \"33028b9510697cbe5c77d92f8f584ffcab23ec66c60730bf4fc12f7c3c6a864a\": rpc error: code = NotFound desc = could not find container \"33028b9510697cbe5c77d92f8f584ffcab23ec66c60730bf4fc12f7c3c6a864a\": container with ID starting with 33028b9510697cbe5c77d92f8f584ffcab23ec66c60730bf4fc12f7c3c6a864a not found: ID does not exist" Nov 03 17:26:25.119065 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:25.118990 4947 scope.go:122] "RemoveContainer" containerID="fda76cd77932b7436280edfeff09a7edebabc200d13dfb1bbcc274088f8be176" Nov 03 17:26:25.119267 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:26:25.119249 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fda76cd77932b7436280edfeff09a7edebabc200d13dfb1bbcc274088f8be176\": container with ID starting with fda76cd77932b7436280edfeff09a7edebabc200d13dfb1bbcc274088f8be176 not found: ID does not exist" containerID="fda76cd77932b7436280edfeff09a7edebabc200d13dfb1bbcc274088f8be176" Nov 03 17:26:25.119318 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:25.119274 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fda76cd77932b7436280edfeff09a7edebabc200d13dfb1bbcc274088f8be176"} err="failed to get container status \"fda76cd77932b7436280edfeff09a7edebabc200d13dfb1bbcc274088f8be176\": rpc error: code = NotFound desc = could not find container \"fda76cd77932b7436280edfeff09a7edebabc200d13dfb1bbcc274088f8be176\": container with ID starting with fda76cd77932b7436280edfeff09a7edebabc200d13dfb1bbcc274088f8be176 not found: ID does not exist" Nov 03 17:26:25.170605 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:25.170574 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn"] Nov 03 17:26:25.183577 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:25.183553 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-44736d-openshift-default-7dcd95f67f-bfnzn"] Nov 03 17:26:26.569110 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:26.569075 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" path="/var/lib/kubelet/pods/048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d/volumes" Nov 03 17:26:30.130485 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:26:30.130453 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:26:37.128423 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:37.128385 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc"] Nov 03 17:26:38.673454 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:38.673418 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg"] Nov 03 17:26:38.673993 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:38.673660 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" podUID="b921c483-84fe-4ae5-8f29-f751b6c80ce5" containerName="maas-api" containerID="cri-o://87292d465e496b99432d0f20141504a8ea4ceb6e314bb41a6db57300208c618f" gracePeriod=30 Nov 03 17:26:39.141983 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.141942 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" Nov 03 17:26:39.198407 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.198368 4947 generic.go:356] "Generic (PLEG): container finished" podID="b921c483-84fe-4ae5-8f29-f751b6c80ce5" containerID="87292d465e496b99432d0f20141504a8ea4ceb6e314bb41a6db57300208c618f" exitCode=0 Nov 03 17:26:39.198614 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.198574 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" event={"ID":"b921c483-84fe-4ae5-8f29-f751b6c80ce5","Type":"ContainerDied","Data":"87292d465e496b99432d0f20141504a8ea4ceb6e314bb41a6db57300208c618f"} Nov 03 17:26:39.198614 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.198610 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" event={"ID":"b921c483-84fe-4ae5-8f29-f751b6c80ce5","Type":"ContainerDied","Data":"4711df3189750fff05b18dcd5a0f3f511df1b0cc64353d8c8177c0811a2b5ada"} Nov 03 17:26:39.198813 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.198630 4947 scope.go:122] "RemoveContainer" containerID="87292d465e496b99432d0f20141504a8ea4ceb6e314bb41a6db57300208c618f" Nov 03 17:26:39.198813 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.198794 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg" Nov 03 17:26:39.212525 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.212502 4947 scope.go:122] "RemoveContainer" containerID="87292d465e496b99432d0f20141504a8ea4ceb6e314bb41a6db57300208c618f" Nov 03 17:26:39.212978 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:26:39.212948 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87292d465e496b99432d0f20141504a8ea4ceb6e314bb41a6db57300208c618f\": container with ID starting with 87292d465e496b99432d0f20141504a8ea4ceb6e314bb41a6db57300208c618f not found: ID does not exist" containerID="87292d465e496b99432d0f20141504a8ea4ceb6e314bb41a6db57300208c618f" Nov 03 17:26:39.213088 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.212985 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87292d465e496b99432d0f20141504a8ea4ceb6e314bb41a6db57300208c618f"} err="failed to get container status \"87292d465e496b99432d0f20141504a8ea4ceb6e314bb41a6db57300208c618f\": rpc error: code = NotFound desc = could not find container \"87292d465e496b99432d0f20141504a8ea4ceb6e314bb41a6db57300208c618f\": container with ID starting with 87292d465e496b99432d0f20141504a8ea4ceb6e314bb41a6db57300208c618f not found: ID does not exist" Nov 03 17:26:39.217982 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.217956 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b921c483-84fe-4ae5-8f29-f751b6c80ce5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b921c483-84fe-4ae5-8f29-f751b6c80ce5-maas-api-tls\") pod \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\" (UID: \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\") " Nov 03 17:26:39.218111 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.218081 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b921c483-84fe-4ae5-8f29-f751b6c80ce5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b921c483-84fe-4ae5-8f29-f751b6c80ce5-metrics-certs\") pod \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\" (UID: \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\") " Nov 03 17:26:39.218289 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.218275 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b921c483-84fe-4ae5-8f29-f751b6c80ce5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b921c483-84fe-4ae5-8f29-f751b6c80ce5-openshift-service-ca\") pod \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\" (UID: \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\") " Nov 03 17:26:39.218370 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.218342 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b921c483-84fe-4ae5-8f29-f751b6c80ce5-kube-api-access-qjkrf\" (UniqueName: \"kubernetes.io/projected/b921c483-84fe-4ae5-8f29-f751b6c80ce5-kube-api-access-qjkrf\") pod \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\" (UID: \"b921c483-84fe-4ae5-8f29-f751b6c80ce5\") " Nov 03 17:26:39.219278 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.219244 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b921c483-84fe-4ae5-8f29-f751b6c80ce5-openshift-service-ca" pod "b921c483-84fe-4ae5-8f29-f751b6c80ce5" (UID: "b921c483-84fe-4ae5-8f29-f751b6c80ce5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:26:39.221225 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.221128 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b921c483-84fe-4ae5-8f29-f751b6c80ce5-metrics-certs" pod "b921c483-84fe-4ae5-8f29-f751b6c80ce5" (UID: "b921c483-84fe-4ae5-8f29-f751b6c80ce5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:26:39.221316 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.221150 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b921c483-84fe-4ae5-8f29-f751b6c80ce5-maas-api-tls" pod "b921c483-84fe-4ae5-8f29-f751b6c80ce5" (UID: "b921c483-84fe-4ae5-8f29-f751b6c80ce5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:26:39.221316 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.221195 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b921c483-84fe-4ae5-8f29-f751b6c80ce5-kube-api-access-qjkrf" pod "b921c483-84fe-4ae5-8f29-f751b6c80ce5" (UID: "b921c483-84fe-4ae5-8f29-f751b6c80ce5"). InnerVolumeSpecName "kube-api-access-qjkrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:39.320458 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.320383 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b921c483-84fe-4ae5-8f29-f751b6c80ce5-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:39.320458 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.320410 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjkrf\" (UniqueName: \"kubernetes.io/projected/b921c483-84fe-4ae5-8f29-f751b6c80ce5-kube-api-access-qjkrf\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:39.320458 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.320420 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b921c483-84fe-4ae5-8f29-f751b6c80ce5-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:39.320458 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.320428 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b921c483-84fe-4ae5-8f29-f751b6c80ce5-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:39.559394 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.559351 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg"] Nov 03 17:26:39.562664 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:39.562642 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-547925-84ff8b7f7-2gpfg"] Nov 03 17:26:40.566839 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:40.566805 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b921c483-84fe-4ae5-8f29-f751b6c80ce5" path="/var/lib/kubelet/pods/b921c483-84fe-4ae5-8f29-f751b6c80ce5/volumes" Nov 03 17:26:44.274874 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:44.274835 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2"] Nov 03 17:26:44.311708 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:44.311668 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg"] Nov 03 17:26:49.276189 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:49.276127 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" podUID="f98e92c6-7bc2-4211-b7c2-524f7eddfe88" containerName="payload-processing" containerID="cri-o://2171c81f0ecd1fbcc9c3c91f6a830dc58d5227a4397a112642096344e14596fc" gracePeriod=25 Nov 03 17:26:49.312250 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:49.312217 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" podUID="baec733a-0e6d-44c7-8807-4f7437806318" containerName="payload-pre-processing" containerID="cri-o://f2a7114627a977bc158ef9c5bde1ae099a65355dd162523f3f9a260a7b3af98e" gracePeriod=25 Nov 03 17:26:52.153045 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.152998 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" podUID="1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9" containerName="main" containerID="cri-o://546cdb81ea451bc8b3eefd4178b7f9d8132b958dc734d9cb308ed6b657ccd438" gracePeriod=45 Nov 03 17:26:52.319952 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.319914 4947 generic.go:356] "Generic (PLEG): container finished" podID="1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9" containerID="546cdb81ea451bc8b3eefd4178b7f9d8132b958dc734d9cb308ed6b657ccd438" exitCode=0 Nov 03 17:26:52.320178 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.319990 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" event={"ID":"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9","Type":"ContainerDied","Data":"546cdb81ea451bc8b3eefd4178b7f9d8132b958dc734d9cb308ed6b657ccd438"} Nov 03 17:26:52.472751 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.472723 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:26:52.560840 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.560174 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-dshm\") pod \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " Nov 03 17:26:52.560840 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.560275 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-home\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-home\") pod \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " Nov 03 17:26:52.560840 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.560430 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-model-cache\") pod \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " Nov 03 17:26:52.560840 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.560501 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-tmp-dir\") pod \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " Nov 03 17:26:52.560840 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.560600 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-kube-api-access-xn8vv\" (UniqueName: \"kubernetes.io/projected/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-kube-api-access-xn8vv\") pod \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " Nov 03 17:26:52.560840 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.560633 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-tls-certs\") pod \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\" (UID: \"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9\") " Nov 03 17:26:52.565028 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.564995 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-model-cache" pod "1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9" (UID: "1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:52.565713 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.565671 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-home" pod "1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9" (UID: "1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:52.574053 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.569946 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-tmp-dir" pod "1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9" (UID: "1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:52.585564 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.585501 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-kube-api-access-xn8vv" pod "1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9" (UID: "1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9"). InnerVolumeSpecName "kube-api-access-xn8vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:26:52.585822 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.585803 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-dshm" pod "1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9" (UID: "1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:26:52.592919 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.592005 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-tls-certs" pod "1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9" (UID: "1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:26:52.665219 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.665187 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:52.665384 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.665223 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn8vv\" (UniqueName: \"kubernetes.io/projected/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-kube-api-access-xn8vv\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:52.665384 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.665240 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:52.665384 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.665254 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:52.665384 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.665267 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:52.665384 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:52.665279 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:26:53.330043 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:53.330013 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" Nov 03 17:26:53.330505 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:53.330056 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" event={"ID":"1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9","Type":"ContainerDied","Data":"7adaefba9c211bd603a869ce73af6607f3784c772b2c32ea67eb07ba65f05594"} Nov 03 17:26:53.330505 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:53.330108 4947 scope.go:122] "RemoveContainer" containerID="546cdb81ea451bc8b3eefd4178b7f9d8132b958dc734d9cb308ed6b657ccd438" Nov 03 17:26:53.427858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:53.427828 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc"] Nov 03 17:26:53.447291 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:53.447264 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc"] Nov 03 17:26:53.466922 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:53.466892 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-89df9a/test-model-89df9a-kserve-85f7f879cc-cj9bc" podUID="1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9" containerName="main" probeResult="failure" output="Get \"https://10.132.1.12:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:26:54.461464 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:54.461423 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" podUID="f98e92c6-7bc2-4211-b7c2-524f7eddfe88" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:26:54.463903 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:54.461435 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" podUID="baec733a-0e6d-44c7-8807-4f7437806318" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:26:54.568378 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:26:54.568336 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9" path="/var/lib/kubelet/pods/1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9/volumes" Nov 03 17:27:00.157905 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:27:00.157803 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:27:04.461693 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:04.461651 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" podUID="baec733a-0e6d-44c7-8807-4f7437806318" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:27:04.461693 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:04.461711 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" podUID="f98e92c6-7bc2-4211-b7c2-524f7eddfe88" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:27:07.416875 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:07.416842 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:27:09.473267 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:09.473232 4947 generic.go:356] "Generic (PLEG): container finished" podID="f98e92c6-7bc2-4211-b7c2-524f7eddfe88" containerID="2171c81f0ecd1fbcc9c3c91f6a830dc58d5227a4397a112642096344e14596fc" exitCode=0 Nov 03 17:27:09.473668 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:09.473390 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" event={"ID":"f98e92c6-7bc2-4211-b7c2-524f7eddfe88","Type":"ContainerDied","Data":"2171c81f0ecd1fbcc9c3c91f6a830dc58d5227a4397a112642096344e14596fc"} Nov 03 17:27:09.475475 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:09.475449 4947 generic.go:356] "Generic (PLEG): container finished" podID="baec733a-0e6d-44c7-8807-4f7437806318" containerID="f2a7114627a977bc158ef9c5bde1ae099a65355dd162523f3f9a260a7b3af98e" exitCode=0 Nov 03 17:27:09.475596 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:09.475581 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" event={"ID":"baec733a-0e6d-44c7-8807-4f7437806318","Type":"ContainerDied","Data":"f2a7114627a977bc158ef9c5bde1ae099a65355dd162523f3f9a260a7b3af98e"} Nov 03 17:27:09.601344 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:09.601044 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" Nov 03 17:27:09.655547 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:09.655518 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" Nov 03 17:27:09.740809 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:09.739391 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98e92c6-7bc2-4211-b7c2-524f7eddfe88-kube-api-access-tv5gr\" (UniqueName: \"kubernetes.io/projected/f98e92c6-7bc2-4211-b7c2-524f7eddfe88-kube-api-access-tv5gr\") pod \"f98e92c6-7bc2-4211-b7c2-524f7eddfe88\" (UID: \"f98e92c6-7bc2-4211-b7c2-524f7eddfe88\") " Nov 03 17:27:09.740809 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:09.739603 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f98e92c6-7bc2-4211-b7c2-524f7eddfe88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98e92c6-7bc2-4211-b7c2-524f7eddfe88-plugins-config-volume\") pod \"f98e92c6-7bc2-4211-b7c2-524f7eddfe88\" (UID: \"f98e92c6-7bc2-4211-b7c2-524f7eddfe88\") " Nov 03 17:27:09.741675 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:09.741647 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98e92c6-7bc2-4211-b7c2-524f7eddfe88-plugins-config-volume" pod "f98e92c6-7bc2-4211-b7c2-524f7eddfe88" (UID: "f98e92c6-7bc2-4211-b7c2-524f7eddfe88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:27:09.744347 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:09.744326 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98e92c6-7bc2-4211-b7c2-524f7eddfe88-kube-api-access-tv5gr" pod "f98e92c6-7bc2-4211-b7c2-524f7eddfe88" (UID: "f98e92c6-7bc2-4211-b7c2-524f7eddfe88"). InnerVolumeSpecName "kube-api-access-tv5gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:09.841266 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:09.841188 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baec733a-0e6d-44c7-8807-4f7437806318-kube-api-access-2jvn2\" (UniqueName: \"kubernetes.io/projected/baec733a-0e6d-44c7-8807-4f7437806318-kube-api-access-2jvn2\") pod \"baec733a-0e6d-44c7-8807-4f7437806318\" (UID: \"baec733a-0e6d-44c7-8807-4f7437806318\") " Nov 03 17:27:09.841266 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:09.841251 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baec733a-0e6d-44c7-8807-4f7437806318-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baec733a-0e6d-44c7-8807-4f7437806318-plugins-config-volume\") pod \"baec733a-0e6d-44c7-8807-4f7437806318\" (UID: \"baec733a-0e6d-44c7-8807-4f7437806318\") " Nov 03 17:27:09.841713 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:09.841680 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv5gr\" (UniqueName: \"kubernetes.io/projected/f98e92c6-7bc2-4211-b7c2-524f7eddfe88-kube-api-access-tv5gr\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:27:09.841846 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:09.841720 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98e92c6-7bc2-4211-b7c2-524f7eddfe88-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:27:09.841846 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:09.841769 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baec733a-0e6d-44c7-8807-4f7437806318-plugins-config-volume" pod "baec733a-0e6d-44c7-8807-4f7437806318" (UID: "baec733a-0e6d-44c7-8807-4f7437806318"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:27:09.843533 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:09.843511 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baec733a-0e6d-44c7-8807-4f7437806318-kube-api-access-2jvn2" pod "baec733a-0e6d-44c7-8807-4f7437806318" (UID: "baec733a-0e6d-44c7-8807-4f7437806318"). InnerVolumeSpecName "kube-api-access-2jvn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:09.943228 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:09.943198 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jvn2\" (UniqueName: \"kubernetes.io/projected/baec733a-0e6d-44c7-8807-4f7437806318-kube-api-access-2jvn2\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:27:09.943228 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:09.943226 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baec733a-0e6d-44c7-8807-4f7437806318-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:27:10.484529 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:10.484495 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" Nov 03 17:27:10.484529 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:10.484509 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2" event={"ID":"f98e92c6-7bc2-4211-b7c2-524f7eddfe88","Type":"ContainerDied","Data":"4778d912408aa913c5a762d8d3713388d8ea483c213bcd81f845f78ed0aa935f"} Nov 03 17:27:10.485135 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:10.484578 4947 scope.go:122] "RemoveContainer" containerID="2171c81f0ecd1fbcc9c3c91f6a830dc58d5227a4397a112642096344e14596fc" Nov 03 17:27:10.486361 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:10.486338 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" Nov 03 17:27:10.486502 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:10.486417 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg" event={"ID":"baec733a-0e6d-44c7-8807-4f7437806318","Type":"ContainerDied","Data":"5de5562ca1aad63dac947eb9203d2378f247b821240b28279529876871c7b946"} Nov 03 17:27:10.496133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:10.496107 4947 scope.go:122] "RemoveContainer" containerID="f2a7114627a977bc158ef9c5bde1ae099a65355dd162523f3f9a260a7b3af98e" Nov 03 17:27:10.548635 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:10.548576 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2"] Nov 03 17:27:10.567990 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:10.567966 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-472bee-66b9649b4c-lzhq2"] Nov 03 17:27:10.588903 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:10.588873 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg"] Nov 03 17:27:10.591757 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:10.591735 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-472bee-6fb957c9b7-249tg"] Nov 03 17:27:12.566865 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:12.566831 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baec733a-0e6d-44c7-8807-4f7437806318" path="/var/lib/kubelet/pods/baec733a-0e6d-44c7-8807-4f7437806318/volumes" Nov 03 17:27:12.567281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:12.567271 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f98e92c6-7bc2-4211-b7c2-524f7eddfe88" path="/var/lib/kubelet/pods/f98e92c6-7bc2-4211-b7c2-524f7eddfe88/volumes" Nov 03 17:27:30.132153 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:27:30.132112 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:27:33.957090 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957059 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm"] Nov 03 17:27:33.957781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957765 4947 cpu_manager.go:414] "containerMap: removing container" podUID="b2938160-d944-4c19-abd6-f634b57b920b" containerName="payload-pre-processing" Nov 03 17:27:33.957837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957782 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2938160-d944-4c19-abd6-f634b57b920b" containerName="payload-pre-processing" Nov 03 17:27:33.957837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957799 4947 cpu_manager.go:414] "containerMap: removing container" podUID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" containerName="istio-proxy" Nov 03 17:27:33.957837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957803 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" containerName="istio-proxy" Nov 03 17:27:33.957837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957810 4947 cpu_manager.go:414] "containerMap: removing container" podUID="1e32622d-7ba0-45ad-a866-05089e15f14e" containerName="maas-api" Nov 03 17:27:33.957837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957815 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e32622d-7ba0-45ad-a866-05089e15f14e" containerName="maas-api" Nov 03 17:27:33.957837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957827 4947 cpu_manager.go:414] "containerMap: removing container" podUID="8c3d1351-007d-4fe5-be27-d868a8655c32" containerName="payload-processing" Nov 03 17:27:33.957837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957831 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3d1351-007d-4fe5-be27-d868a8655c32" containerName="payload-processing" Nov 03 17:27:33.957837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957839 4947 cpu_manager.go:414] "containerMap: removing container" podUID="e7c02310-bff9-4d86-b641-8022d891b395" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957845 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7c02310-bff9-4d86-b641-8022d891b395" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957852 4947 cpu_manager.go:414] "containerMap: removing container" podUID="23ddeb2f-f94f-4012-a987-259a185e19aa" containerName="maas-api" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957856 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23ddeb2f-f94f-4012-a987-259a185e19aa" containerName="maas-api" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957862 4947 cpu_manager.go:414] "containerMap: removing container" podUID="f98e92c6-7bc2-4211-b7c2-524f7eddfe88" containerName="payload-processing" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957866 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98e92c6-7bc2-4211-b7c2-524f7eddfe88" containerName="payload-processing" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957873 4947 cpu_manager.go:414] "containerMap: removing container" podUID="2d4fef6b-5ca6-41c1-8809-a74e3c3401e1" containerName="main" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957877 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d4fef6b-5ca6-41c1-8809-a74e3c3401e1" containerName="main" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957893 4947 cpu_manager.go:414] "containerMap: removing container" podUID="407a82a9-162d-4120-82a1-f6022279dced" containerName="payload-pre-processing" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957897 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="407a82a9-162d-4120-82a1-f6022279dced" containerName="payload-pre-processing" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957906 4947 cpu_manager.go:414] "containerMap: removing container" podUID="2ead42d5-f3f0-4e06-a578-d4d457639850" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957911 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ead42d5-f3f0-4e06-a578-d4d457639850" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957920 4947 cpu_manager.go:414] "containerMap: removing container" podUID="1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9" containerName="main" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957925 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9" containerName="main" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957935 4947 cpu_manager.go:414] "containerMap: removing container" podUID="ea91c101-fb9f-4277-83d0-39229045f116" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957939 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea91c101-fb9f-4277-83d0-39229045f116" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957945 4947 cpu_manager.go:414] "containerMap: removing container" podUID="83746e7d-ec09-4142-b46b-38fa1471cc4a" containerName="payload-pre-processing" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957950 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83746e7d-ec09-4142-b46b-38fa1471cc4a" containerName="payload-pre-processing" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957956 4947 cpu_manager.go:414] "containerMap: removing container" podUID="769d68a5-7504-4c6a-9db1-355bff4c7ef8" containerName="maas-api" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957960 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769d68a5-7504-4c6a-9db1-355bff4c7ef8" containerName="maas-api" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957973 4947 cpu_manager.go:414] "containerMap: removing container" podUID="629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957977 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957984 4947 cpu_manager.go:414] "containerMap: removing container" podUID="0fc48229-09ca-4ae5-9cb2-de8e8ea21cad" containerName="maas-api" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957988 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc48229-09ca-4ae5-9cb2-de8e8ea21cad" containerName="maas-api" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.957999 4947 cpu_manager.go:414] "containerMap: removing container" podUID="d9f0e126-f94a-4cb6-b187-e05f23476696" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958004 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9f0e126-f94a-4cb6-b187-e05f23476696" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958011 4947 cpu_manager.go:414] "containerMap: removing container" podUID="c7a45866-e8e8-4731-b688-4674bf874a44" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958015 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a45866-e8e8-4731-b688-4674bf874a44" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958024 4947 cpu_manager.go:414] "containerMap: removing container" podUID="54adffe4-51ee-41e0-8d1b-9d329330d512" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958028 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54adffe4-51ee-41e0-8d1b-9d329330d512" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958034 4947 cpu_manager.go:414] "containerMap: removing container" podUID="1513cf28-6652-4788-a4eb-215a9cf87a77" containerName="maas-api" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958039 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1513cf28-6652-4788-a4eb-215a9cf87a77" containerName="maas-api" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958046 4947 cpu_manager.go:414] "containerMap: removing container" podUID="e2b50359-b597-46ac-9ca1-af75415e1cf1" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958050 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2b50359-b597-46ac-9ca1-af75415e1cf1" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958057 4947 cpu_manager.go:414] "containerMap: removing container" podUID="c7a45866-e8e8-4731-b688-4674bf874a44" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958061 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a45866-e8e8-4731-b688-4674bf874a44" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958071 4947 cpu_manager.go:414] "containerMap: removing container" podUID="1508d21f-1858-4232-be82-bcda731862b9" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958076 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1508d21f-1858-4232-be82-bcda731862b9" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958086 4947 cpu_manager.go:414] "containerMap: removing container" podUID="b921c483-84fe-4ae5-8f29-f751b6c80ce5" containerName="maas-api" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958090 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b921c483-84fe-4ae5-8f29-f751b6c80ce5" containerName="maas-api" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958095 4947 cpu_manager.go:414] "containerMap: removing container" podUID="6318894b-7e77-47fa-805e-a28d53c1fead" containerName="main" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958099 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6318894b-7e77-47fa-805e-a28d53c1fead" containerName="main" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958106 4947 cpu_manager.go:414] "containerMap: removing container" podUID="a852b387-ca23-4e7a-b793-788feb393176" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958110 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a852b387-ca23-4e7a-b793-788feb393176" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958117 4947 cpu_manager.go:414] "containerMap: removing container" podUID="cfcceda1-486c-48fe-a25e-d7d37f97e5a3" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958120 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfcceda1-486c-48fe-a25e-d7d37f97e5a3" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958128 4947 cpu_manager.go:414] "containerMap: removing container" podUID="93035580-fe13-49e9-8c14-4878b6763fbd" containerName="main" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958132 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93035580-fe13-49e9-8c14-4878b6763fbd" containerName="main" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958137 4947 cpu_manager.go:414] "containerMap: removing container" podUID="1508d21f-1858-4232-be82-bcda731862b9" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958141 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1508d21f-1858-4232-be82-bcda731862b9" containerName="istio-proxy" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958152 4947 cpu_manager.go:414] "containerMap: removing container" podUID="baec733a-0e6d-44c7-8807-4f7437806318" containerName="payload-pre-processing" Nov 03 17:27:33.958136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958156 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baec733a-0e6d-44c7-8807-4f7437806318" containerName="payload-pre-processing" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958165 4947 cpu_manager.go:414] "containerMap: removing container" podUID="458486ff-035b-44c6-853c-c6b071e5370f" containerName="payload-pre-processing" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958170 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="458486ff-035b-44c6-853c-c6b071e5370f" containerName="payload-pre-processing" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958176 4947 cpu_manager.go:414] "containerMap: removing container" podUID="a852b387-ca23-4e7a-b793-788feb393176" containerName="istio-proxy" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958180 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a852b387-ca23-4e7a-b793-788feb393176" containerName="istio-proxy" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958272 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a45866-e8e8-4731-b688-4674bf874a44" containerName="istio-proxy" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958282 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="1508d21f-1858-4232-be82-bcda731862b9" containerName="istio-proxy" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958287 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="b921c483-84fe-4ae5-8f29-f751b6c80ce5" containerName="maas-api" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958293 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6318894b-7e77-47fa-805e-a28d53c1fead" containerName="main" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958299 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="a852b387-ca23-4e7a-b793-788feb393176" containerName="istio-proxy" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958304 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfcceda1-486c-48fe-a25e-d7d37f97e5a3" containerName="istio-proxy" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958311 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="93035580-fe13-49e9-8c14-4878b6763fbd" containerName="main" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958317 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="1508d21f-1858-4232-be82-bcda731862b9" containerName="istio-proxy" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958323 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="baec733a-0e6d-44c7-8807-4f7437806318" containerName="payload-pre-processing" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958329 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="458486ff-035b-44c6-853c-c6b071e5370f" containerName="payload-pre-processing" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958334 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="a852b387-ca23-4e7a-b793-788feb393176" containerName="istio-proxy" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958339 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2938160-d944-4c19-abd6-f634b57b920b" containerName="payload-pre-processing" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958345 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" containerName="istio-proxy" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958350 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e32622d-7ba0-45ad-a866-05089e15f14e" containerName="maas-api" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958356 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3d1351-007d-4fe5-be27-d868a8655c32" containerName="payload-processing" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958362 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7c02310-bff9-4d86-b641-8022d891b395" containerName="istio-proxy" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958366 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="23ddeb2f-f94f-4012-a987-259a185e19aa" containerName="maas-api" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958371 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98e92c6-7bc2-4211-b7c2-524f7eddfe88" containerName="payload-processing" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958375 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d4fef6b-5ca6-41c1-8809-a74e3c3401e1" containerName="main" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958381 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ead42d5-f3f0-4e06-a578-d4d457639850" containerName="istio-proxy" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958385 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="407a82a9-162d-4120-82a1-f6022279dced" containerName="payload-pre-processing" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958392 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f6b355b-00f5-4ca1-8d73-4e5c7ca51cf9" containerName="main" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958399 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea91c101-fb9f-4277-83d0-39229045f116" containerName="istio-proxy" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958404 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="83746e7d-ec09-4142-b46b-38fa1471cc4a" containerName="payload-pre-processing" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958409 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="769d68a5-7504-4c6a-9db1-355bff4c7ef8" containerName="maas-api" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958415 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="629a8dc0-af1f-4e22-aeaa-e2fa86a4a009" containerName="istio-proxy" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958421 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc48229-09ca-4ae5-9cb2-de8e8ea21cad" containerName="maas-api" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958426 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9f0e126-f94a-4cb6-b187-e05f23476696" containerName="istio-proxy" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958432 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="54adffe4-51ee-41e0-8d1b-9d329330d512" containerName="istio-proxy" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958442 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" containerName="istio-proxy" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958447 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="1513cf28-6652-4788-a4eb-215a9cf87a77" containerName="maas-api" Nov 03 17:27:33.960368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.958452 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2b50359-b597-46ac-9ca1-af75415e1cf1" containerName="istio-proxy" Nov 03 17:27:33.962409 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:33.962390 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.003759 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.003723 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm"] Nov 03 17:27:34.007997 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.007963 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.008178 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.008058 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.008178 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.008132 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.008460 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.008197 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.008460 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.008225 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.008460 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.008251 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.008460 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.008320 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.008460 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.008382 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.008460 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.008423 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssf95\" (UniqueName: \"kubernetes.io/projected/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-kube-api-access-ssf95\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.109549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.109506 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.109799 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.109608 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.109799 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.109675 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.109939 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.109807 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.109939 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.109859 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.109939 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.109929 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.110103 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.109944 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.110103 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.110003 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.110103 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.110038 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssf95\" (UniqueName: \"kubernetes.io/projected/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-kube-api-access-ssf95\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.110313 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.110133 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.110472 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.110450 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.110904 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.110883 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.111018 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.110884 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.111153 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.111133 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.112779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.112752 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.113298 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.113275 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.141811 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.141792 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.142096 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.142075 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssf95\" (UniqueName: \"kubernetes.io/projected/3b61ea26-243b-4e8a-8ca7-3a154a72a9b7-kube-api-access-ssf95\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-9gxvm\" (UID: \"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.277544 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.277517 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:34.637866 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.637823 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm"] Nov 03 17:27:34.645359 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.645322 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:27:34.645500 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.645404 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:27:34.645500 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.645449 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:27:34.650268 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:34.650242 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" event={"ID":"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7","Type":"ContainerStarted","Data":"31da93dadc8955b8b85f194700dfa856f2d68eaf55995fa7386d9b100e88e3a2"} Nov 03 17:27:35.659416 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:35.659379 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" event={"ID":"3b61ea26-243b-4e8a-8ca7-3a154a72a9b7","Type":"ContainerStarted","Data":"18d76ef96952f5c97cdb37b238aed5181e04283902c1e6c2a07b6cd893e72399"} Nov 03 17:27:35.729286 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:35.729222 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" podStartSLOduration=2.729202575 podStartE2EDuration="2.729202575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:27:35.727131173 +0000 UTC m=+4399.316131142" watchObservedRunningTime="2026-11-03 17:27:35.729202575 +0000 UTC m=+4399.318202543" Nov 03 17:27:36.277944 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:36.277910 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:36.279708 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:36.279675 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Nov 03 17:27:36.279862 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:36.279756 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" podUID="3b61ea26-243b-4e8a-8ca7-3a154a72a9b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Nov 03 17:27:37.278430 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:37.278391 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Nov 03 17:27:37.278976 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:37.278470 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" podUID="3b61ea26-243b-4e8a-8ca7-3a154a72a9b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Nov 03 17:27:38.278345 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:38.278307 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Nov 03 17:27:38.278535 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:38.278360 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" podUID="3b61ea26-243b-4e8a-8ca7-3a154a72a9b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Nov 03 17:27:39.278906 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:39.278874 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Nov 03 17:27:39.279322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:39.278924 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" podUID="3b61ea26-243b-4e8a-8ca7-3a154a72a9b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Nov 03 17:27:40.278226 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:40.278189 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Nov 03 17:27:40.278435 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:40.278249 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" podUID="3b61ea26-243b-4e8a-8ca7-3a154a72a9b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Nov 03 17:27:41.278737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:41.278680 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Nov 03 17:27:41.279148 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:41.278765 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" podUID="3b61ea26-243b-4e8a-8ca7-3a154a72a9b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Nov 03 17:27:42.278770 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:42.278733 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Nov 03 17:27:42.279183 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:42.278786 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" podUID="3b61ea26-243b-4e8a-8ca7-3a154a72a9b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Nov 03 17:27:43.314931 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:43.314888 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7"] Nov 03 17:27:43.316505 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:43.316474 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" podUID="0fc32f2f-8c40-45dd-b7cb-d93125ada2af" containerName="istio-proxy" containerID="cri-o://2d1394ba36af7770926b44553f1334dc89d27162f4139a16d8fbf2dba6d94597" gracePeriod=30 Nov 03 17:27:43.378270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:43.378235 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:43.379299 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:43.379271 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:43.381823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:43.381797 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" Nov 03 17:27:45.051471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.051435 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9"] Nov 03 17:27:45.052447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.052417 4947 cpu_manager.go:414] "containerMap: removing container" podUID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" containerName="istio-proxy" Nov 03 17:27:45.052447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.052443 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048bfe2e-c1d7-42c1-a75b-0bf2a25cb33d" containerName="istio-proxy" Nov 03 17:27:45.052655 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.052591 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a45866-e8e8-4731-b688-4674bf874a44" containerName="istio-proxy" Nov 03 17:27:45.058139 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.058110 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.093242 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.093212 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-06fa9cd7-openshift-default-dockercfg-rzrs7\"" Nov 03 17:27:45.147947 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.147910 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9"] Nov 03 17:27:45.252637 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.252602 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-workload-certs\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.253051 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.252668 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a74f616-c592-4caa-9918-b774a56814af-istiod-ca-cert\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.253051 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.252842 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-istio-envoy\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.253051 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.252937 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptqq2\" (UniqueName: \"kubernetes.io/projected/6a74f616-c592-4caa-9918-b774a56814af-kube-api-access-ptqq2\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.253393 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.253074 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-credential-socket\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.253393 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.253131 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a74f616-c592-4caa-9918-b774a56814af-istio-token\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.253393 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.253302 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a74f616-c592-4caa-9918-b774a56814af-istio-podinfo\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.253393 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.253356 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-istio-data\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.253393 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.253392 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-workload-socket\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.355106 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.355019 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a74f616-c592-4caa-9918-b774a56814af-istio-podinfo\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.355106 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.355070 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-istio-data\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.355106 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.355098 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-workload-socket\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.355432 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.355129 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-workload-certs\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.355432 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.355163 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a74f616-c592-4caa-9918-b774a56814af-istiod-ca-cert\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.355432 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.355221 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-istio-envoy\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.355432 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.355318 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptqq2\" (UniqueName: \"kubernetes.io/projected/6a74f616-c592-4caa-9918-b774a56814af-kube-api-access-ptqq2\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.355432 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.355403 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-credential-socket\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.355823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.355455 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a74f616-c592-4caa-9918-b774a56814af-istio-token\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.355823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.355561 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-workload-socket\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.355823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.355601 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-istio-data\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.356006 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.355837 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-workload-certs\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.356006 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.355857 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-credential-socket\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.356627 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.356286 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a74f616-c592-4caa-9918-b774a56814af-istiod-ca-cert\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.357907 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.357881 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-istio-envoy\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.358031 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.357914 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a74f616-c592-4caa-9918-b774a56814af-istio-podinfo\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.383902 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.383879 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a74f616-c592-4caa-9918-b774a56814af-istio-token\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.385497 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.385464 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptqq2\" (UniqueName: \"kubernetes.io/projected/6a74f616-c592-4caa-9918-b774a56814af-kube-api-access-ptqq2\") pod \"e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:45.681552 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:45.681467 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:46.017429 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:46.017402 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9"] Nov 03 17:27:46.019462 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:27:46.019425 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a74f616_c592_4caa_9918_b774a56814af.slice/crio-307798dd8f6805ed8415707580e36a405a499c01460a66791e48dff237a00c05 WatchSource:0}: Error finding container 307798dd8f6805ed8415707580e36a405a499c01460a66791e48dff237a00c05: Status 404 returned error can't find the container with id 307798dd8f6805ed8415707580e36a405a499c01460a66791e48dff237a00c05 Nov 03 17:27:46.022544 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:46.022504 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:27:46.022662 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:46.022592 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:27:46.022662 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:46.022638 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:27:46.746294 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:46.746257 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" event={"ID":"6a74f616-c592-4caa-9918-b774a56814af","Type":"ContainerStarted","Data":"3c1be9ffad043cb343dda78bacb39f45a269709113283270b63745cc6a07c59b"} Nov 03 17:27:46.746848 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:46.746323 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" event={"ID":"6a74f616-c592-4caa-9918-b774a56814af","Type":"ContainerStarted","Data":"307798dd8f6805ed8415707580e36a405a499c01460a66791e48dff237a00c05"} Nov 03 17:27:46.803069 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:46.803005 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" podStartSLOduration=1.802986626 podStartE2EDuration="1.802986626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:27:46.798309982 +0000 UTC m=+4410.387309951" watchObservedRunningTime="2026-11-03 17:27:46.802986626 +0000 UTC m=+4410.391986597" Nov 03 17:27:47.389505 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:47.389464 4947 patch_prober.go:29] interesting pod/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:27:47.389719 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:47.389536 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" podUID="0fc32f2f-8c40-45dd-b7cb-d93125ada2af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:27:47.682287 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:47.682209 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:47.688498 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:47.688470 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:47.752736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:47.752702 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:47.753839 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:47.753802 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:27:48.617175 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.617150 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:27:48.703271 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.703240 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istiod-ca-cert\") pod \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " Nov 03 17:27:48.703463 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.703324 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-workload-socket\") pod \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " Nov 03 17:27:48.703463 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.703371 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-data\") pod \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " Nov 03 17:27:48.703463 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.703417 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-workload-certs\") pod \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " Nov 03 17:27:48.704523 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.703465 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-envoy\") pod \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " Nov 03 17:27:48.704523 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.703582 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-kube-api-access-4svzn\" (UniqueName: \"kubernetes.io/projected/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-kube-api-access-4svzn\") pod \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " Nov 03 17:27:48.704523 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.703638 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-token\" (UniqueName: \"kubernetes.io/projected/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-token\") pod \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " Nov 03 17:27:48.704523 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.703688 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-podinfo\") pod \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " Nov 03 17:27:48.704523 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.703869 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-credential-socket\") pod \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\" (UID: \"0fc32f2f-8c40-45dd-b7cb-d93125ada2af\") " Nov 03 17:27:48.705877 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.705850 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-credential-socket" pod "0fc32f2f-8c40-45dd-b7cb-d93125ada2af" (UID: "0fc32f2f-8c40-45dd-b7cb-d93125ada2af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.706339 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.706319 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istiod-ca-cert" pod "0fc32f2f-8c40-45dd-b7cb-d93125ada2af" (UID: "0fc32f2f-8c40-45dd-b7cb-d93125ada2af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:27:48.706730 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.706688 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-workload-socket" pod "0fc32f2f-8c40-45dd-b7cb-d93125ada2af" (UID: "0fc32f2f-8c40-45dd-b7cb-d93125ada2af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.707048 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.707020 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-data" pod "0fc32f2f-8c40-45dd-b7cb-d93125ada2af" (UID: "0fc32f2f-8c40-45dd-b7cb-d93125ada2af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.707219 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.707198 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-workload-certs" pod "0fc32f2f-8c40-45dd-b7cb-d93125ada2af" (UID: "0fc32f2f-8c40-45dd-b7cb-d93125ada2af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.711822 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.711794 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-podinfo" pod "0fc32f2f-8c40-45dd-b7cb-d93125ada2af" (UID: "0fc32f2f-8c40-45dd-b7cb-d93125ada2af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:27:48.713084 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.713062 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-envoy" pod "0fc32f2f-8c40-45dd-b7cb-d93125ada2af" (UID: "0fc32f2f-8c40-45dd-b7cb-d93125ada2af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:27:48.713172 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.713127 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-token" pod "0fc32f2f-8c40-45dd-b7cb-d93125ada2af" (UID: "0fc32f2f-8c40-45dd-b7cb-d93125ada2af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:48.713923 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.713902 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-kube-api-access-4svzn" pod "0fc32f2f-8c40-45dd-b7cb-d93125ada2af" (UID: "0fc32f2f-8c40-45dd-b7cb-d93125ada2af"). InnerVolumeSpecName "kube-api-access-4svzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:27:48.764867 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.764829 4947 generic.go:356] "Generic (PLEG): container finished" podID="0fc32f2f-8c40-45dd-b7cb-d93125ada2af" containerID="2d1394ba36af7770926b44553f1334dc89d27162f4139a16d8fbf2dba6d94597" exitCode=0 Nov 03 17:27:48.765339 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.764892 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" Nov 03 17:27:48.765339 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.764933 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" event={"ID":"0fc32f2f-8c40-45dd-b7cb-d93125ada2af","Type":"ContainerDied","Data":"2d1394ba36af7770926b44553f1334dc89d27162f4139a16d8fbf2dba6d94597"} Nov 03 17:27:48.765339 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.765007 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7" event={"ID":"0fc32f2f-8c40-45dd-b7cb-d93125ada2af","Type":"ContainerDied","Data":"785c859272007136b5e70f49f13a1453a6929f4394a815964e1f6dbc8f4c8d64"} Nov 03 17:27:48.765339 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.765042 4947 scope.go:122] "RemoveContainer" containerID="2d1394ba36af7770926b44553f1334dc89d27162f4139a16d8fbf2dba6d94597" Nov 03 17:27:48.781826 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.781804 4947 scope.go:122] "RemoveContainer" containerID="2d1394ba36af7770926b44553f1334dc89d27162f4139a16d8fbf2dba6d94597" Nov 03 17:27:48.782259 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:27:48.782228 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d1394ba36af7770926b44553f1334dc89d27162f4139a16d8fbf2dba6d94597\": container with ID starting with 2d1394ba36af7770926b44553f1334dc89d27162f4139a16d8fbf2dba6d94597 not found: ID does not exist" containerID="2d1394ba36af7770926b44553f1334dc89d27162f4139a16d8fbf2dba6d94597" Nov 03 17:27:48.782371 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.782271 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1394ba36af7770926b44553f1334dc89d27162f4139a16d8fbf2dba6d94597"} err="failed to get container status \"2d1394ba36af7770926b44553f1334dc89d27162f4139a16d8fbf2dba6d94597\": rpc error: code = NotFound desc = could not find container \"2d1394ba36af7770926b44553f1334dc89d27162f4139a16d8fbf2dba6d94597\": container with ID starting with 2d1394ba36af7770926b44553f1334dc89d27162f4139a16d8fbf2dba6d94597 not found: ID does not exist" Nov 03 17:27:48.806977 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.806946 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.806977 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.806977 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.807229 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.806986 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4svzn\" (UniqueName: \"kubernetes.io/projected/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-kube-api-access-4svzn\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.807229 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.806998 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.807229 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.807007 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.807229 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.807016 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.807229 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.807024 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.807229 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.807032 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.807229 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.807039 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fc32f2f-8c40-45dd-b7cb-d93125ada2af-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:27:48.822878 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.822847 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7"] Nov 03 17:27:48.826473 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:48.826445 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-89df9a-openshift-default-6fd7f78c9-7p8s7"] Nov 03 17:27:50.567493 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:27:50.567461 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc32f2f-8c40-45dd-b7cb-d93125ada2af" path="/var/lib/kubelet/pods/0fc32f2f-8c40-45dd-b7cb-d93125ada2af/volumes" Nov 03 17:28:00.130182 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:28:00.130139 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:28:13.429796 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:13.429762 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:28:27.311086 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.311001 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794"] Nov 03 17:28:27.311752 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.311736 4947 cpu_manager.go:414] "containerMap: removing container" podUID="0fc32f2f-8c40-45dd-b7cb-d93125ada2af" containerName="istio-proxy" Nov 03 17:28:27.311822 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.311752 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc32f2f-8c40-45dd-b7cb-d93125ada2af" containerName="istio-proxy" Nov 03 17:28:27.311899 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.311890 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc32f2f-8c40-45dd-b7cb-d93125ada2af" containerName="istio-proxy" Nov 03 17:28:27.315485 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.315460 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" Nov 03 17:28:27.331993 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.331966 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-06fa9cd7-serving-cert\"" Nov 03 17:28:27.361552 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.361519 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794"] Nov 03 17:28:27.434091 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.434056 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-openshift-service-ca\") pod \"maas-api-e2e-mt-06fa9cd7-5858887f46-km794\" (UID: \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" Nov 03 17:28:27.434266 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.434123 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pljhm\" (UniqueName: \"kubernetes.io/projected/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-kube-api-access-pljhm\") pod \"maas-api-e2e-mt-06fa9cd7-5858887f46-km794\" (UID: \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" Nov 03 17:28:27.434266 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.434236 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-metrics-certs\") pod \"maas-api-e2e-mt-06fa9cd7-5858887f46-km794\" (UID: \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" Nov 03 17:28:27.434366 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.434299 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-maas-api-tls\") pod \"maas-api-e2e-mt-06fa9cd7-5858887f46-km794\" (UID: \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" Nov 03 17:28:27.535899 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.535853 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-metrics-certs\") pod \"maas-api-e2e-mt-06fa9cd7-5858887f46-km794\" (UID: \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" Nov 03 17:28:27.536117 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.535964 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-maas-api-tls\") pod \"maas-api-e2e-mt-06fa9cd7-5858887f46-km794\" (UID: \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" Nov 03 17:28:27.536192 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.536146 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-openshift-service-ca\") pod \"maas-api-e2e-mt-06fa9cd7-5858887f46-km794\" (UID: \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" Nov 03 17:28:27.536243 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.536203 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pljhm\" (UniqueName: \"kubernetes.io/projected/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-kube-api-access-pljhm\") pod \"maas-api-e2e-mt-06fa9cd7-5858887f46-km794\" (UID: \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" Nov 03 17:28:27.537129 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.537104 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-openshift-service-ca\") pod \"maas-api-e2e-mt-06fa9cd7-5858887f46-km794\" (UID: \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" Nov 03 17:28:27.539291 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.539267 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-maas-api-tls\") pod \"maas-api-e2e-mt-06fa9cd7-5858887f46-km794\" (UID: \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" Nov 03 17:28:27.539445 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.539422 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-metrics-certs\") pod \"maas-api-e2e-mt-06fa9cd7-5858887f46-km794\" (UID: \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" Nov 03 17:28:27.560976 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.560945 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pljhm\" (UniqueName: \"kubernetes.io/projected/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-kube-api-access-pljhm\") pod \"maas-api-e2e-mt-06fa9cd7-5858887f46-km794\" (UID: \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" Nov 03 17:28:27.631274 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.631201 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" Nov 03 17:28:27.939030 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:28:27.938999 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb7fe1d_f91b_4ef4_8f5b_aa8577c264c8.slice/crio-4755a68dfe07e219ad1925631d40487bffc184b0d478d54b4d844229b0de33a2 WatchSource:0}: Error finding container 4755a68dfe07e219ad1925631d40487bffc184b0d478d54b4d844229b0de33a2: Status 404 returned error can't find the container with id 4755a68dfe07e219ad1925631d40487bffc184b0d478d54b4d844229b0de33a2 Nov 03 17:28:27.939519 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.939496 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794"] Nov 03 17:28:27.940451 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:27.940432 4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:28:28.053985 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:28.053947 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" event={"ID":"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8","Type":"ContainerStarted","Data":"4755a68dfe07e219ad1925631d40487bffc184b0d478d54b4d844229b0de33a2"} Nov 03 17:28:29.061725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:29.061671 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" event={"ID":"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8","Type":"ContainerStarted","Data":"b885ff33d43edec6b26ba54fab5b4a8dbe20e94a2a674eb5ae4cdfa78d93b08c"} Nov 03 17:28:29.062200 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:29.061829 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" Nov 03 17:28:29.111513 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:29.111464 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" podStartSLOduration=1.62644506 podStartE2EDuration="2.111450913s" totalImagesPullingTime="485.005853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:28:29.107728882 +0000 UTC m=+4452.696728841" watchObservedRunningTime="2026-11-03 17:28:29.111450913 +0000 UTC m=+4452.700450880" Nov 03 17:28:30.134033 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:28:30.133991 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:28:34.124227 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.124195 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf"] Nov 03 17:28:34.129260 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.129234 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.160544 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.160517 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf"] Nov 03 17:28:34.161878 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.161844 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.162013 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.161969 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.162069 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.162034 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.162572 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.162119 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvnb8\" (UniqueName: \"kubernetes.io/projected/64013461-1f82-44ce-9c7e-6dfc9d4b9159-kube-api-access-xvnb8\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.162572 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.162196 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.162572 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.162254 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.162572 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.162278 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.162572 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.162321 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.162572 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.162344 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.263543 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.263510 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvnb8\" (UniqueName: \"kubernetes.io/projected/64013461-1f82-44ce-9c7e-6dfc9d4b9159-kube-api-access-xvnb8\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.263740 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.263613 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.263740 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.263680 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.263846 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.263754 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.263846 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.263813 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.263952 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.263856 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.263952 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.263924 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.264068 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.263975 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.264068 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.264007 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.264187 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.264078 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.264246 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.264177 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.264285 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.264246 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.264442 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.264417 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.264572 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.264519 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.266466 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.266443 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.266604 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.266585 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.288686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.288650 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.288857 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.288789 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvnb8\" (UniqueName: \"kubernetes.io/projected/64013461-1f82-44ce-9c7e-6dfc9d4b9159-kube-api-access-xvnb8\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-z7szf\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.445064 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.444983 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:34.795334 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:28:34.795296 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64013461_1f82_44ce_9c7e_6dfc9d4b9159.slice/crio-5527e745e6636f35476472418d8766cd7cc0d7aa36fbd0aac81bb6da8bdd16a4 WatchSource:0}: Error finding container 5527e745e6636f35476472418d8766cd7cc0d7aa36fbd0aac81bb6da8bdd16a4: Status 404 returned error can't find the container with id 5527e745e6636f35476472418d8766cd7cc0d7aa36fbd0aac81bb6da8bdd16a4 Nov 03 17:28:34.796370 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.796349 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf"] Nov 03 17:28:34.798455 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.798422 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:28:34.798530 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.798485 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:28:34.798530 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:34.798523 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:28:35.075940 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:35.075860 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" Nov 03 17:28:35.104856 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:35.104825 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" event={"ID":"64013461-1f82-44ce-9c7e-6dfc9d4b9159","Type":"ContainerStarted","Data":"a9f95c224bfabdde8020502c316fffcd2d74e7f536f6f0e34f44dfafa2e90a80"} Nov 03 17:28:35.105057 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:35.104876 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" event={"ID":"64013461-1f82-44ce-9c7e-6dfc9d4b9159","Type":"ContainerStarted","Data":"5527e745e6636f35476472418d8766cd7cc0d7aa36fbd0aac81bb6da8bdd16a4"} Nov 03 17:28:35.175511 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:35.175447 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" podStartSLOduration=1.175428212 podStartE2EDuration="1.175428212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:28:35.172632765 +0000 UTC m=+4458.761632734" watchObservedRunningTime="2026-11-03 17:28:35.175428212 +0000 UTC m=+4458.764428181" Nov 03 17:28:35.445924 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:35.445840 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:35.447750 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:35.447727 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z7szf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Nov 03 17:28:35.447830 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:35.447781 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" podUID="64013461-1f82-44ce-9c7e-6dfc9d4b9159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Nov 03 17:28:36.464723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:36.451163 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z7szf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Nov 03 17:28:36.464723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:36.451244 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" podUID="64013461-1f82-44ce-9c7e-6dfc9d4b9159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Nov 03 17:28:37.446073 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:37.446033 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z7szf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Nov 03 17:28:37.446242 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:37.446093 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" podUID="64013461-1f82-44ce-9c7e-6dfc9d4b9159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Nov 03 17:28:38.445387 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:38.445349 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z7szf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Nov 03 17:28:38.445890 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:38.445412 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" podUID="64013461-1f82-44ce-9c7e-6dfc9d4b9159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Nov 03 17:28:39.447314 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:39.447272 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z7szf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Nov 03 17:28:39.447879 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:39.447346 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" podUID="64013461-1f82-44ce-9c7e-6dfc9d4b9159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Nov 03 17:28:40.446199 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:40.446164 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z7szf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Nov 03 17:28:40.446362 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:40.446225 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" podUID="64013461-1f82-44ce-9c7e-6dfc9d4b9159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Nov 03 17:28:41.445936 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:41.445899 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-z7szf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" start-of-body= Nov 03 17:28:41.446413 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:41.445957 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" podUID="64013461-1f82-44ce-9c7e-6dfc9d4b9159" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.16:15021/healthz/ready\": dial tcp 10.132.1.16:15021: connect: connection refused" Nov 03 17:28:43.274835 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:43.274798 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:43.276023 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:43.276000 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:43.276918 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:43.276897 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:28:48.440819 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:28:48.440785 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:29:00.132420 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:29:00.132331 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:29:00.298910 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:00.298873 4947 generic.go:356] "Generic (PLEG): container finished" podID="6a74f616-c592-4caa-9918-b774a56814af" containerID="3c1be9ffad043cb343dda78bacb39f45a269709113283270b63745cc6a07c59b" exitCode=0 Nov 03 17:29:00.299107 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:00.298923 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" event={"ID":"6a74f616-c592-4caa-9918-b774a56814af","Type":"ContainerDied","Data":"3c1be9ffad043cb343dda78bacb39f45a269709113283270b63745cc6a07c59b"} Nov 03 17:29:00.299486 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:00.299459 4947 scope.go:122] "RemoveContainer" containerID="3c1be9ffad043cb343dda78bacb39f45a269709113283270b63745cc6a07c59b" Nov 03 17:29:00.301658 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:00.301626 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:29:00.301779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:00.301715 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:29:00.301779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:00.301751 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:29:01.330111 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:01.330069 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" event={"ID":"6a74f616-c592-4caa-9918-b774a56814af","Type":"ContainerStarted","Data":"f63d19213c2f35c65d339c67ecce2127f681b643d2a8bde4e99a509ed6e8d338"} Nov 03 17:29:01.682549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:01.682474 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:29:01.683889 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:01.683867 4947 patch_prober.go:29] interesting pod/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Nov 03 17:29:01.683953 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:01.683923 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" podUID="6a74f616-c592-4caa-9918-b774a56814af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Nov 03 17:29:02.682321 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:02.682287 4947 patch_prober.go:29] interesting pod/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Nov 03 17:29:02.682805 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:02.682377 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" podUID="6a74f616-c592-4caa-9918-b774a56814af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Nov 03 17:29:02.752666 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:02.752626 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:29:03.682495 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:03.682455 4947 patch_prober.go:29] interesting pod/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Nov 03 17:29:03.683015 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:03.682518 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" podUID="6a74f616-c592-4caa-9918-b774a56814af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Nov 03 17:29:04.682461 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:04.682426 4947 patch_prober.go:29] interesting pod/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Nov 03 17:29:04.682645 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:04.682478 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" podUID="6a74f616-c592-4caa-9918-b774a56814af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Nov 03 17:29:05.681869 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:05.681828 4947 patch_prober.go:29] interesting pod/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Nov 03 17:29:05.682183 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:05.681889 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" podUID="6a74f616-c592-4caa-9918-b774a56814af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Nov 03 17:29:06.687073 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:06.687044 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:29:06.688159 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:06.688138 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:29:10.768893 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:10.768856 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l"] Nov 03 17:29:10.802909 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:10.802873 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm"] Nov 03 17:29:15.770104 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:15.770038 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" podUID="d6fe77c5-ca27-418f-971d-842b755b3e5a" containerName="payload-processing" containerID="cri-o://f39f898350221526e5db4fe81c5da892965d7e4d0e89dd07535d32c83b932620" gracePeriod=25 Nov 03 17:29:15.803408 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:15.803363 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" podUID="65d3f051-43e5-43a7-a879-5a14425498db" containerName="payload-pre-processing" containerID="cri-o://cf6cdce8dab586b2a17cd0c58f46001d1109aef4b46358e6819623377d83d0eb" gracePeriod=25 Nov 03 17:29:17.315212 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:17.315180 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:29:17.321784 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:17.321760 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:29:21.696587 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:21.696545 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" podUID="d6fe77c5-ca27-418f-971d-842b755b3e5a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:29:21.697050 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:21.696638 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" podUID="65d3f051-43e5-43a7-a879-5a14425498db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:29:25.781429 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:25.781394 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:29:28.086050 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.086019 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d"] Nov 03 17:29:28.086510 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.086204 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" podUID="afd191dc-3ff4-4451-b93a-da68f606c84e" containerName="maas-api" containerID="cri-o://be183b1d7c29d9451bac33d2f99dbb254aac4226e86122290c7d93eb848a6320" gracePeriod=30 Nov 03 17:29:28.451018 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.450987 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:29:28.559725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.556830 4947 generic.go:356] "Generic (PLEG): container finished" podID="afd191dc-3ff4-4451-b93a-da68f606c84e" containerID="be183b1d7c29d9451bac33d2f99dbb254aac4226e86122290c7d93eb848a6320" exitCode=0 Nov 03 17:29:28.559725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.556944 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" event={"ID":"afd191dc-3ff4-4451-b93a-da68f606c84e","Type":"ContainerDied","Data":"be183b1d7c29d9451bac33d2f99dbb254aac4226e86122290c7d93eb848a6320"} Nov 03 17:29:28.559725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.557024 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" event={"ID":"afd191dc-3ff4-4451-b93a-da68f606c84e","Type":"ContainerDied","Data":"86b82a57ee851ea308f4f340c32ace4bded030dea9717c10b96b3bc4a9968774"} Nov 03 17:29:28.559725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.557051 4947 scope.go:122] "RemoveContainer" containerID="be183b1d7c29d9451bac33d2f99dbb254aac4226e86122290c7d93eb848a6320" Nov 03 17:29:28.559725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.557192 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d" Nov 03 17:29:28.634179 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.634058 4947 scope.go:122] "RemoveContainer" containerID="be183b1d7c29d9451bac33d2f99dbb254aac4226e86122290c7d93eb848a6320" Nov 03 17:29:28.634718 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:29:28.634629 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be183b1d7c29d9451bac33d2f99dbb254aac4226e86122290c7d93eb848a6320\": container with ID starting with be183b1d7c29d9451bac33d2f99dbb254aac4226e86122290c7d93eb848a6320 not found: ID does not exist" containerID="be183b1d7c29d9451bac33d2f99dbb254aac4226e86122290c7d93eb848a6320" Nov 03 17:29:28.634718 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.634665 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be183b1d7c29d9451bac33d2f99dbb254aac4226e86122290c7d93eb848a6320"} err="failed to get container status \"be183b1d7c29d9451bac33d2f99dbb254aac4226e86122290c7d93eb848a6320\": rpc error: code = NotFound desc = could not find container \"be183b1d7c29d9451bac33d2f99dbb254aac4226e86122290c7d93eb848a6320\": container with ID starting with be183b1d7c29d9451bac33d2f99dbb254aac4226e86122290c7d93eb848a6320 not found: ID does not exist" Nov 03 17:29:28.636595 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.636573 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afd191dc-3ff4-4451-b93a-da68f606c84e-kube-api-access-g9x8m\" (UniqueName: \"kubernetes.io/projected/afd191dc-3ff4-4451-b93a-da68f606c84e-kube-api-access-g9x8m\") pod \"afd191dc-3ff4-4451-b93a-da68f606c84e\" (UID: \"afd191dc-3ff4-4451-b93a-da68f606c84e\") " Nov 03 17:29:28.636846 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.636828 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afd191dc-3ff4-4451-b93a-da68f606c84e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd191dc-3ff4-4451-b93a-da68f606c84e-openshift-service-ca\") pod \"afd191dc-3ff4-4451-b93a-da68f606c84e\" (UID: \"afd191dc-3ff4-4451-b93a-da68f606c84e\") " Nov 03 17:29:28.636984 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.636969 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afd191dc-3ff4-4451-b93a-da68f606c84e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd191dc-3ff4-4451-b93a-da68f606c84e-maas-api-tls\") pod \"afd191dc-3ff4-4451-b93a-da68f606c84e\" (UID: \"afd191dc-3ff4-4451-b93a-da68f606c84e\") " Nov 03 17:29:28.637211 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.637196 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afd191dc-3ff4-4451-b93a-da68f606c84e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd191dc-3ff4-4451-b93a-da68f606c84e-metrics-certs\") pod \"afd191dc-3ff4-4451-b93a-da68f606c84e\" (UID: \"afd191dc-3ff4-4451-b93a-da68f606c84e\") " Nov 03 17:29:28.639726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.639683 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afd191dc-3ff4-4451-b93a-da68f606c84e-openshift-service-ca" pod "afd191dc-3ff4-4451-b93a-da68f606c84e" (UID: "afd191dc-3ff4-4451-b93a-da68f606c84e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:29:28.659885 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.659822 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd191dc-3ff4-4451-b93a-da68f606c84e-maas-api-tls" pod "afd191dc-3ff4-4451-b93a-da68f606c84e" (UID: "afd191dc-3ff4-4451-b93a-da68f606c84e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:29:28.660536 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.660503 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afd191dc-3ff4-4451-b93a-da68f606c84e-metrics-certs" pod "afd191dc-3ff4-4451-b93a-da68f606c84e" (UID: "afd191dc-3ff4-4451-b93a-da68f606c84e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:29:28.670337 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.670258 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afd191dc-3ff4-4451-b93a-da68f606c84e-kube-api-access-g9x8m" pod "afd191dc-3ff4-4451-b93a-da68f606c84e" (UID: "afd191dc-3ff4-4451-b93a-da68f606c84e"). InnerVolumeSpecName "kube-api-access-g9x8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:29:28.739322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.739290 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/afd191dc-3ff4-4451-b93a-da68f606c84e-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:29:28.739322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.739318 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/afd191dc-3ff4-4451-b93a-da68f606c84e-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:29:28.739322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.739329 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/afd191dc-3ff4-4451-b93a-da68f606c84e-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:29:28.739549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.739336 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9x8m\" (UniqueName: \"kubernetes.io/projected/afd191dc-3ff4-4451-b93a-da68f606c84e-kube-api-access-g9x8m\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:29:28.920136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.920103 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d"] Nov 03 17:29:28.936589 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:28.936526 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1f83c0-5d956cf475-9nj7d"] Nov 03 17:29:30.130581 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:29:30.130541 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:29:30.566264 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:30.566231 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afd191dc-3ff4-4451-b93a-da68f606c84e" path="/var/lib/kubelet/pods/afd191dc-3ff4-4451-b93a-da68f606c84e/volumes" Nov 03 17:29:31.697531 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:31.697492 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" podUID="65d3f051-43e5-43a7-a879-5a14425498db" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:29:31.697531 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:31.697517 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" podUID="d6fe77c5-ca27-418f-971d-842b755b3e5a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:29:36.148545 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.148511 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" Nov 03 17:29:36.200338 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.200304 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" Nov 03 17:29:36.282168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.282132 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65d3f051-43e5-43a7-a879-5a14425498db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65d3f051-43e5-43a7-a879-5a14425498db-plugins-config-volume\") pod \"65d3f051-43e5-43a7-a879-5a14425498db\" (UID: \"65d3f051-43e5-43a7-a879-5a14425498db\") " Nov 03 17:29:36.282336 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.282198 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6fe77c5-ca27-418f-971d-842b755b3e5a-kube-api-access-twrf4\" (UniqueName: \"kubernetes.io/projected/d6fe77c5-ca27-418f-971d-842b755b3e5a-kube-api-access-twrf4\") pod \"d6fe77c5-ca27-418f-971d-842b755b3e5a\" (UID: \"d6fe77c5-ca27-418f-971d-842b755b3e5a\") " Nov 03 17:29:36.282336 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.282234 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65d3f051-43e5-43a7-a879-5a14425498db-kube-api-access-4fxnb\" (UniqueName: \"kubernetes.io/projected/65d3f051-43e5-43a7-a879-5a14425498db-kube-api-access-4fxnb\") pod \"65d3f051-43e5-43a7-a879-5a14425498db\" (UID: \"65d3f051-43e5-43a7-a879-5a14425498db\") " Nov 03 17:29:36.282465 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.282349 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6fe77c5-ca27-418f-971d-842b755b3e5a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6fe77c5-ca27-418f-971d-842b755b3e5a-plugins-config-volume\") pod \"d6fe77c5-ca27-418f-971d-842b755b3e5a\" (UID: \"d6fe77c5-ca27-418f-971d-842b755b3e5a\") " Nov 03 17:29:36.282813 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.282753 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d3f051-43e5-43a7-a879-5a14425498db-plugins-config-volume" pod "65d3f051-43e5-43a7-a879-5a14425498db" (UID: "65d3f051-43e5-43a7-a879-5a14425498db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:29:36.282813 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.282794 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6fe77c5-ca27-418f-971d-842b755b3e5a-plugins-config-volume" pod "d6fe77c5-ca27-418f-971d-842b755b3e5a" (UID: "d6fe77c5-ca27-418f-971d-842b755b3e5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:29:36.283095 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.283068 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65d3f051-43e5-43a7-a879-5a14425498db-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:29:36.283095 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.283097 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6fe77c5-ca27-418f-971d-842b755b3e5a-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:29:36.285637 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.285616 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6fe77c5-ca27-418f-971d-842b755b3e5a-kube-api-access-twrf4" pod "d6fe77c5-ca27-418f-971d-842b755b3e5a" (UID: "d6fe77c5-ca27-418f-971d-842b755b3e5a"). InnerVolumeSpecName "kube-api-access-twrf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:29:36.285786 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.285644 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d3f051-43e5-43a7-a879-5a14425498db-kube-api-access-4fxnb" pod "65d3f051-43e5-43a7-a879-5a14425498db" (UID: "65d3f051-43e5-43a7-a879-5a14425498db"). InnerVolumeSpecName "kube-api-access-4fxnb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:29:36.384659 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.384580 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twrf4\" (UniqueName: \"kubernetes.io/projected/d6fe77c5-ca27-418f-971d-842b755b3e5a-kube-api-access-twrf4\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:29:36.384659 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.384611 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fxnb\" (UniqueName: \"kubernetes.io/projected/65d3f051-43e5-43a7-a879-5a14425498db-kube-api-access-4fxnb\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:29:36.642949 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.642866 4947 generic.go:356] "Generic (PLEG): container finished" podID="65d3f051-43e5-43a7-a879-5a14425498db" containerID="cf6cdce8dab586b2a17cd0c58f46001d1109aef4b46358e6819623377d83d0eb" exitCode=0 Nov 03 17:29:36.642949 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.642920 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" Nov 03 17:29:36.643168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.642951 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" event={"ID":"65d3f051-43e5-43a7-a879-5a14425498db","Type":"ContainerDied","Data":"cf6cdce8dab586b2a17cd0c58f46001d1109aef4b46358e6819623377d83d0eb"} Nov 03 17:29:36.643168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.643005 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm" event={"ID":"65d3f051-43e5-43a7-a879-5a14425498db","Type":"ContainerDied","Data":"bf4ac4bfe0082532d1a3fd726ef0d1051622521e5d9f3006698e920e5697a48f"} Nov 03 17:29:36.643168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.643036 4947 scope.go:122] "RemoveContainer" containerID="cf6cdce8dab586b2a17cd0c58f46001d1109aef4b46358e6819623377d83d0eb" Nov 03 17:29:36.644722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.644680 4947 generic.go:356] "Generic (PLEG): container finished" podID="d6fe77c5-ca27-418f-971d-842b755b3e5a" containerID="f39f898350221526e5db4fe81c5da892965d7e4d0e89dd07535d32c83b932620" exitCode=0 Nov 03 17:29:36.644950 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.644729 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" event={"ID":"d6fe77c5-ca27-418f-971d-842b755b3e5a","Type":"ContainerDied","Data":"f39f898350221526e5db4fe81c5da892965d7e4d0e89dd07535d32c83b932620"} Nov 03 17:29:36.645065 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.644979 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" event={"ID":"d6fe77c5-ca27-418f-971d-842b755b3e5a","Type":"ContainerDied","Data":"0127231d5b3c56669230d0045e96ce5738cd1d55261cf09c50c4e26978342750"} Nov 03 17:29:36.645065 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.644766 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l" Nov 03 17:29:36.656100 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.656082 4947 scope.go:122] "RemoveContainer" containerID="cf6cdce8dab586b2a17cd0c58f46001d1109aef4b46358e6819623377d83d0eb" Nov 03 17:29:36.656408 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:29:36.656375 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6cdce8dab586b2a17cd0c58f46001d1109aef4b46358e6819623377d83d0eb\": container with ID starting with cf6cdce8dab586b2a17cd0c58f46001d1109aef4b46358e6819623377d83d0eb not found: ID does not exist" containerID="cf6cdce8dab586b2a17cd0c58f46001d1109aef4b46358e6819623377d83d0eb" Nov 03 17:29:36.656537 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.656407 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6cdce8dab586b2a17cd0c58f46001d1109aef4b46358e6819623377d83d0eb"} err="failed to get container status \"cf6cdce8dab586b2a17cd0c58f46001d1109aef4b46358e6819623377d83d0eb\": rpc error: code = NotFound desc = could not find container \"cf6cdce8dab586b2a17cd0c58f46001d1109aef4b46358e6819623377d83d0eb\": container with ID starting with cf6cdce8dab586b2a17cd0c58f46001d1109aef4b46358e6819623377d83d0eb not found: ID does not exist" Nov 03 17:29:36.656537 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.656424 4947 scope.go:122] "RemoveContainer" containerID="f39f898350221526e5db4fe81c5da892965d7e4d0e89dd07535d32c83b932620" Nov 03 17:29:36.667369 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.667352 4947 scope.go:122] "RemoveContainer" containerID="f39f898350221526e5db4fe81c5da892965d7e4d0e89dd07535d32c83b932620" Nov 03 17:29:36.667670 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:29:36.667649 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39f898350221526e5db4fe81c5da892965d7e4d0e89dd07535d32c83b932620\": container with ID starting with f39f898350221526e5db4fe81c5da892965d7e4d0e89dd07535d32c83b932620 not found: ID does not exist" containerID="f39f898350221526e5db4fe81c5da892965d7e4d0e89dd07535d32c83b932620" Nov 03 17:29:36.667768 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.667678 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39f898350221526e5db4fe81c5da892965d7e4d0e89dd07535d32c83b932620"} err="failed to get container status \"f39f898350221526e5db4fe81c5da892965d7e4d0e89dd07535d32c83b932620\": rpc error: code = NotFound desc = could not find container \"f39f898350221526e5db4fe81c5da892965d7e4d0e89dd07535d32c83b932620\": container with ID starting with f39f898350221526e5db4fe81c5da892965d7e4d0e89dd07535d32c83b932620 not found: ID does not exist" Nov 03 17:29:36.701857 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.701833 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l"] Nov 03 17:29:36.714271 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.714247 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1f83c0-6f6fd46669-tzp7l"] Nov 03 17:29:36.743542 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.743506 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm"] Nov 03 17:29:36.750883 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:36.750852 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1f83c0-75645dc9f5-96wgm"] Nov 03 17:29:38.567445 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:38.567410 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65d3f051-43e5-43a7-a879-5a14425498db" path="/var/lib/kubelet/pods/65d3f051-43e5-43a7-a879-5a14425498db/volumes" Nov 03 17:29:38.567958 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:29:38.567887 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6fe77c5-ca27-418f-971d-842b755b3e5a" path="/var/lib/kubelet/pods/d6fe77c5-ca27-418f-971d-842b755b3e5a/volumes" Nov 03 17:30:00.130217 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:00.130137 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:30:00.233742 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.233687 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv"] Nov 03 17:30:00.235229 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.235202 4947 cpu_manager.go:414] "containerMap: removing container" podUID="65d3f051-43e5-43a7-a879-5a14425498db" containerName="payload-pre-processing" Nov 03 17:30:00.235229 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.235229 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65d3f051-43e5-43a7-a879-5a14425498db" containerName="payload-pre-processing" Nov 03 17:30:00.235439 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.235246 4947 cpu_manager.go:414] "containerMap: removing container" podUID="d6fe77c5-ca27-418f-971d-842b755b3e5a" containerName="payload-processing" Nov 03 17:30:00.235439 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.235253 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6fe77c5-ca27-418f-971d-842b755b3e5a" containerName="payload-processing" Nov 03 17:30:00.235439 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.235303 4947 cpu_manager.go:414] "containerMap: removing container" podUID="afd191dc-3ff4-4451-b93a-da68f606c84e" containerName="maas-api" Nov 03 17:30:00.235439 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.235311 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afd191dc-3ff4-4451-b93a-da68f606c84e" containerName="maas-api" Nov 03 17:30:00.235646 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.235493 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="65d3f051-43e5-43a7-a879-5a14425498db" containerName="payload-pre-processing" Nov 03 17:30:00.235646 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.235511 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6fe77c5-ca27-418f-971d-842b755b3e5a" containerName="payload-processing" Nov 03 17:30:00.235646 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.235528 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="afd191dc-3ff4-4451-b93a-da68f606c84e" containerName="maas-api" Nov 03 17:30:00.239959 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.239936 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7"] Nov 03 17:30:00.240135 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.240082 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" Nov 03 17:30:00.246551 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.246305 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" Nov 03 17:30:00.250688 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.250636 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc"] Nov 03 17:30:00.255885 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.255862 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" Nov 03 17:30:00.262409 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.262385 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv"] Nov 03 17:30:00.269899 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.269871 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7"] Nov 03 17:30:00.283831 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.283805 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt"] Nov 03 17:30:00.307743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.293508 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7q7j9\"" Nov 03 17:30:00.307743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.294171 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" Nov 03 17:30:00.307743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.299342 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n"] Nov 03 17:30:00.307743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.305183 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc"] Nov 03 17:30:00.307743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.305476 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wscd5\" (UniqueName: \"kubernetes.io/projected/01cb2467-7a13-4f96-9c67-2cc5104d99b9-kube-api-access-wscd5\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv\" (UID: \"01cb2467-7a13-4f96-9c67-2cc5104d99b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" Nov 03 17:30:00.307743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.307668 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l68hl\" (UniqueName: \"kubernetes.io/projected/aa47da19-7ea1-4601-bfba-18eca9086932-kube-api-access-l68hl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7\" (UID: \"aa47da19-7ea1-4601-bfba-18eca9086932\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" Nov 03 17:30:00.308269 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.307762 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01cb2467-7a13-4f96-9c67-2cc5104d99b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv\" (UID: \"01cb2467-7a13-4f96-9c67-2cc5104d99b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" Nov 03 17:30:00.308269 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.307849 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53960065-d46f-4926-82c0-6b180e9eab56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc\" (UID: \"53960065-d46f-4926-82c0-6b180e9eab56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" Nov 03 17:30:00.308269 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.307879 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmkrd\" (UniqueName: \"kubernetes.io/projected/53960065-d46f-4926-82c0-6b180e9eab56-kube-api-access-kmkrd\") pod \"maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc\" (UID: \"53960065-d46f-4926-82c0-6b180e9eab56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" Nov 03 17:30:00.308269 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.307925 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa47da19-7ea1-4601-bfba-18eca9086932-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7\" (UID: \"aa47da19-7ea1-4601-bfba-18eca9086932\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" Nov 03 17:30:00.308269 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.305587 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" Nov 03 17:30:00.327609 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.327580 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn"] Nov 03 17:30:00.347736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.342236 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt"] Nov 03 17:30:00.347736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.342283 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6"] Nov 03 17:30:00.347736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.345829 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" Nov 03 17:30:00.354511 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.354488 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n"] Nov 03 17:30:00.354640 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.354522 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6"] Nov 03 17:30:00.354640 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.354588 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" Nov 03 17:30:00.355094 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.355066 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn"] Nov 03 17:30:00.408953 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.408887 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqbwb\" (UniqueName: \"kubernetes.io/projected/bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7-kube-api-access-cqbwb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt\" (UID: \"bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" Nov 03 17:30:00.409072 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.409023 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af508d5-80f2-4f5a-b960-bbc2d604e97f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6\" (UID: \"5af508d5-80f2-4f5a-b960-bbc2d604e97f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" Nov 03 17:30:00.409072 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.409058 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wscd5\" (UniqueName: \"kubernetes.io/projected/01cb2467-7a13-4f96-9c67-2cc5104d99b9-kube-api-access-wscd5\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv\" (UID: \"01cb2467-7a13-4f96-9c67-2cc5104d99b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" Nov 03 17:30:00.409149 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.409087 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l68hl\" (UniqueName: \"kubernetes.io/projected/aa47da19-7ea1-4601-bfba-18eca9086932-kube-api-access-l68hl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7\" (UID: \"aa47da19-7ea1-4601-bfba-18eca9086932\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" Nov 03 17:30:00.409185 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.409156 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt\" (UID: \"bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" Nov 03 17:30:00.409239 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.409223 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01cb2467-7a13-4f96-9c67-2cc5104d99b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv\" (UID: \"01cb2467-7a13-4f96-9c67-2cc5104d99b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" Nov 03 17:30:00.409287 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.409262 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c92bab9-7a09-4cc8-b4a9-51bd85bfd818-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn\" (UID: \"0c92bab9-7a09-4cc8-b4a9-51bd85bfd818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" Nov 03 17:30:00.409385 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.409365 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9644e7c0-4d23-498e-8a40-e76349d2e468-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n\" (UID: \"9644e7c0-4d23-498e-8a40-e76349d2e468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" Nov 03 17:30:00.409455 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.409408 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7lf4\" (UniqueName: \"kubernetes.io/projected/5af508d5-80f2-4f5a-b960-bbc2d604e97f-kube-api-access-w7lf4\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6\" (UID: \"5af508d5-80f2-4f5a-b960-bbc2d604e97f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" Nov 03 17:30:00.409529 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.409506 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drvwk\" (UniqueName: \"kubernetes.io/projected/0c92bab9-7a09-4cc8-b4a9-51bd85bfd818-kube-api-access-drvwk\") pod \"maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn\" (UID: \"0c92bab9-7a09-4cc8-b4a9-51bd85bfd818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" Nov 03 17:30:00.409567 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.409556 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53960065-d46f-4926-82c0-6b180e9eab56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc\" (UID: \"53960065-d46f-4926-82c0-6b180e9eab56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" Nov 03 17:30:00.409603 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.409587 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmkrd\" (UniqueName: \"kubernetes.io/projected/53960065-d46f-4926-82c0-6b180e9eab56-kube-api-access-kmkrd\") pod \"maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc\" (UID: \"53960065-d46f-4926-82c0-6b180e9eab56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" Nov 03 17:30:00.409645 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.409615 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxvtk\" (UniqueName: \"kubernetes.io/projected/9644e7c0-4d23-498e-8a40-e76349d2e468-kube-api-access-pxvtk\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n\" (UID: \"9644e7c0-4d23-498e-8a40-e76349d2e468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" Nov 03 17:30:00.409692 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.409663 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa47da19-7ea1-4601-bfba-18eca9086932-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7\" (UID: \"aa47da19-7ea1-4601-bfba-18eca9086932\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" Nov 03 17:30:00.410027 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.410006 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01cb2467-7a13-4f96-9c67-2cc5104d99b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv\" (UID: \"01cb2467-7a13-4f96-9c67-2cc5104d99b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" Nov 03 17:30:00.410182 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.410164 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53960065-d46f-4926-82c0-6b180e9eab56-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc\" (UID: \"53960065-d46f-4926-82c0-6b180e9eab56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" Nov 03 17:30:00.410298 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.410282 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa47da19-7ea1-4601-bfba-18eca9086932-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7\" (UID: \"aa47da19-7ea1-4601-bfba-18eca9086932\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" Nov 03 17:30:00.433916 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.433892 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmkrd\" (UniqueName: \"kubernetes.io/projected/53960065-d46f-4926-82c0-6b180e9eab56-kube-api-access-kmkrd\") pod \"maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc\" (UID: \"53960065-d46f-4926-82c0-6b180e9eab56\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" Nov 03 17:30:00.434241 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.434220 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wscd5\" (UniqueName: \"kubernetes.io/projected/01cb2467-7a13-4f96-9c67-2cc5104d99b9-kube-api-access-wscd5\") pod \"maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv\" (UID: \"01cb2467-7a13-4f96-9c67-2cc5104d99b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" Nov 03 17:30:00.434292 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.434220 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l68hl\" (UniqueName: \"kubernetes.io/projected/aa47da19-7ea1-4601-bfba-18eca9086932-kube-api-access-l68hl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7\" (UID: \"aa47da19-7ea1-4601-bfba-18eca9086932\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" Nov 03 17:30:00.510560 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.510531 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqbwb\" (UniqueName: \"kubernetes.io/projected/bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7-kube-api-access-cqbwb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt\" (UID: \"bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" Nov 03 17:30:00.510729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.510603 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af508d5-80f2-4f5a-b960-bbc2d604e97f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6\" (UID: \"5af508d5-80f2-4f5a-b960-bbc2d604e97f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" Nov 03 17:30:00.510729 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.510648 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt\" (UID: \"bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" Nov 03 17:30:00.510841 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.510731 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c92bab9-7a09-4cc8-b4a9-51bd85bfd818-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn\" (UID: \"0c92bab9-7a09-4cc8-b4a9-51bd85bfd818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" Nov 03 17:30:00.510907 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.510845 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9644e7c0-4d23-498e-8a40-e76349d2e468-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n\" (UID: \"9644e7c0-4d23-498e-8a40-e76349d2e468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" Nov 03 17:30:00.510907 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.510894 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7lf4\" (UniqueName: \"kubernetes.io/projected/5af508d5-80f2-4f5a-b960-bbc2d604e97f-kube-api-access-w7lf4\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6\" (UID: \"5af508d5-80f2-4f5a-b960-bbc2d604e97f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" Nov 03 17:30:00.511018 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.510942 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drvwk\" (UniqueName: \"kubernetes.io/projected/0c92bab9-7a09-4cc8-b4a9-51bd85bfd818-kube-api-access-drvwk\") pod \"maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn\" (UID: \"0c92bab9-7a09-4cc8-b4a9-51bd85bfd818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" Nov 03 17:30:00.511080 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.511043 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxvtk\" (UniqueName: \"kubernetes.io/projected/9644e7c0-4d23-498e-8a40-e76349d2e468-kube-api-access-pxvtk\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n\" (UID: \"9644e7c0-4d23-498e-8a40-e76349d2e468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" Nov 03 17:30:00.511440 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.511415 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af508d5-80f2-4f5a-b960-bbc2d604e97f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6\" (UID: \"5af508d5-80f2-4f5a-b960-bbc2d604e97f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" Nov 03 17:30:00.511440 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.511430 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c92bab9-7a09-4cc8-b4a9-51bd85bfd818-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn\" (UID: \"0c92bab9-7a09-4cc8-b4a9-51bd85bfd818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" Nov 03 17:30:00.511600 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.511447 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt\" (UID: \"bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" Nov 03 17:30:00.511600 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.511550 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9644e7c0-4d23-498e-8a40-e76349d2e468-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n\" (UID: \"9644e7c0-4d23-498e-8a40-e76349d2e468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" Nov 03 17:30:00.544174 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.544140 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7lf4\" (UniqueName: \"kubernetes.io/projected/5af508d5-80f2-4f5a-b960-bbc2d604e97f-kube-api-access-w7lf4\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6\" (UID: \"5af508d5-80f2-4f5a-b960-bbc2d604e97f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" Nov 03 17:30:00.544342 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.544231 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqbwb\" (UniqueName: \"kubernetes.io/projected/bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7-kube-api-access-cqbwb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt\" (UID: \"bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" Nov 03 17:30:00.544342 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.544320 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxvtk\" (UniqueName: \"kubernetes.io/projected/9644e7c0-4d23-498e-8a40-e76349d2e468-kube-api-access-pxvtk\") pod \"maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n\" (UID: \"9644e7c0-4d23-498e-8a40-e76349d2e468\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" Nov 03 17:30:00.544998 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.544975 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drvwk\" (UniqueName: \"kubernetes.io/projected/0c92bab9-7a09-4cc8-b4a9-51bd85bfd818-kube-api-access-drvwk\") pod \"maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn\" (UID: \"0c92bab9-7a09-4cc8-b4a9-51bd85bfd818\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" Nov 03 17:30:00.568349 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.568320 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" Nov 03 17:30:00.610002 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.609965 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" Nov 03 17:30:00.613413 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.613046 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" Nov 03 17:30:00.650481 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.648628 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" Nov 03 17:30:00.662329 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.662081 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" Nov 03 17:30:00.669575 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.669386 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" Nov 03 17:30:00.677079 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.676588 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" Nov 03 17:30:00.903637 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:00.903608 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv"] Nov 03 17:30:00.905675 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:30:00.905508 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01cb2467_7a13_4f96_9c67_2cc5104d99b9.slice/crio-8d61e922f45464c6d1653f203f9573bb6a97be7a7ad76a23361f6a7466491805 WatchSource:0}: Error finding container 8d61e922f45464c6d1653f203f9573bb6a97be7a7ad76a23361f6a7466491805: Status 404 returned error can't find the container with id 8d61e922f45464c6d1653f203f9573bb6a97be7a7ad76a23361f6a7466491805 Nov 03 17:30:01.102268 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.102237 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn"] Nov 03 17:30:01.102687 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:30:01.102666 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c92bab9_7a09_4cc8_b4a9_51bd85bfd818.slice/crio-ebd7cfb51fe861ffaed1f39d6aaf01cdf43c1917dc50fe520a35234110b1e24a WatchSource:0}: Error finding container ebd7cfb51fe861ffaed1f39d6aaf01cdf43c1917dc50fe520a35234110b1e24a: Status 404 returned error can't find the container with id ebd7cfb51fe861ffaed1f39d6aaf01cdf43c1917dc50fe520a35234110b1e24a Nov 03 17:30:01.133912 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.133885 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt"] Nov 03 17:30:01.137747 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.137689 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n"] Nov 03 17:30:01.139286 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:30:01.139263 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9644e7c0_4d23_498e_8a40_e76349d2e468.slice/crio-a5980bd56bc0f73693ee73f4eb070f2178407d8af050789f9e053e13e4266436 WatchSource:0}: Error finding container a5980bd56bc0f73693ee73f4eb070f2178407d8af050789f9e053e13e4266436: Status 404 returned error can't find the container with id a5980bd56bc0f73693ee73f4eb070f2178407d8af050789f9e053e13e4266436 Nov 03 17:30:01.141048 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:30:01.141024 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbe6d1c1_8bd2_44b4_af60_02a3fe977aa7.slice/crio-52663dfa9f5ace73bcacb4ba99fa40e610af9dc87d3090400687ec67f130dbed WatchSource:0}: Error finding container 52663dfa9f5ace73bcacb4ba99fa40e610af9dc87d3090400687ec67f130dbed: Status 404 returned error can't find the container with id 52663dfa9f5ace73bcacb4ba99fa40e610af9dc87d3090400687ec67f130dbed Nov 03 17:30:01.141267 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.141249 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7"] Nov 03 17:30:01.141393 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:30:01.141374 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa47da19_7ea1_4601_bfba_18eca9086932.slice/crio-f5f5a0b39ac61e3fcc72533f37a3b852201e8bc36eabb51937322f626c4b8c6b WatchSource:0}: Error finding container f5f5a0b39ac61e3fcc72533f37a3b852201e8bc36eabb51937322f626c4b8c6b: Status 404 returned error can't find the container with id f5f5a0b39ac61e3fcc72533f37a3b852201e8bc36eabb51937322f626c4b8c6b Nov 03 17:30:01.174590 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.174557 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc"] Nov 03 17:30:01.187139 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.187111 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6"] Nov 03 17:30:01.891686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.891647 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" event={"ID":"9644e7c0-4d23-498e-8a40-e76349d2e468","Type":"ContainerStarted","Data":"b4751f6670fa2510e7c8651ed86157267fcd856f75164c243303f76a408fc4d2"} Nov 03 17:30:01.891893 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.891756 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" event={"ID":"9644e7c0-4d23-498e-8a40-e76349d2e468","Type":"ContainerStarted","Data":"a5980bd56bc0f73693ee73f4eb070f2178407d8af050789f9e053e13e4266436"} Nov 03 17:30:01.893261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.893223 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" event={"ID":"bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7","Type":"ContainerStarted","Data":"400e51361cf378ddf3c35438b3b3f970c7c051f909423e8c83ff4060c6a9398f"} Nov 03 17:30:01.893416 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.893280 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" event={"ID":"bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7","Type":"ContainerStarted","Data":"52663dfa9f5ace73bcacb4ba99fa40e610af9dc87d3090400687ec67f130dbed"} Nov 03 17:30:01.894863 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.894834 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" event={"ID":"5af508d5-80f2-4f5a-b960-bbc2d604e97f","Type":"ContainerStarted","Data":"67bcba1ae1deb1161971408d6e77114efe3ba1cc04e198d31ede595c052e108a"} Nov 03 17:30:01.895190 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.894883 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" event={"ID":"5af508d5-80f2-4f5a-b960-bbc2d604e97f","Type":"ContainerStarted","Data":"c776946c0e28099be55c54388afcbce46e0ec92de2415daea74d526c7be2fbb0"} Nov 03 17:30:01.897202 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.897179 4947 generic.go:356] "Generic (PLEG): container finished" podID="01cb2467-7a13-4f96-9c67-2cc5104d99b9" containerID="fc7391cf191cc0e0bdc4bb3ce28302cacf106f3a6d50d568c42a427b485ae520" exitCode=60 Nov 03 17:30:01.897321 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.897254 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" event={"ID":"01cb2467-7a13-4f96-9c67-2cc5104d99b9","Type":"ContainerDied","Data":"fc7391cf191cc0e0bdc4bb3ce28302cacf106f3a6d50d568c42a427b485ae520"} Nov 03 17:30:01.897321 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.897293 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" event={"ID":"01cb2467-7a13-4f96-9c67-2cc5104d99b9","Type":"ContainerStarted","Data":"8d61e922f45464c6d1653f203f9573bb6a97be7a7ad76a23361f6a7466491805"} Nov 03 17:30:01.897424 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.897409 4947 scope.go:122] "RemoveContainer" containerID="fc7391cf191cc0e0bdc4bb3ce28302cacf106f3a6d50d568c42a427b485ae520" Nov 03 17:30:01.902996 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.902971 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" event={"ID":"53960065-d46f-4926-82c0-6b180e9eab56","Type":"ContainerStarted","Data":"4dd1d2db6a81f22a859aee5739d122980a3a9c8127cb50f442d91cdb610ff2ee"} Nov 03 17:30:01.903113 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.903017 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" event={"ID":"53960065-d46f-4926-82c0-6b180e9eab56","Type":"ContainerStarted","Data":"e545ce802ace0c249c7b661a05f6e497cae5767887de4763f26c8404ea217001"} Nov 03 17:30:01.906329 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.906303 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" event={"ID":"0c92bab9-7a09-4cc8-b4a9-51bd85bfd818","Type":"ContainerStarted","Data":"79907f02daeb2b15be8cc2e499a309b7d8eb7a34d87bf9a2c32980fbbcf59a58"} Nov 03 17:30:01.906431 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.906348 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" event={"ID":"0c92bab9-7a09-4cc8-b4a9-51bd85bfd818","Type":"ContainerStarted","Data":"ebd7cfb51fe861ffaed1f39d6aaf01cdf43c1917dc50fe520a35234110b1e24a"} Nov 03 17:30:01.907784 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.907762 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" event={"ID":"aa47da19-7ea1-4601-bfba-18eca9086932","Type":"ContainerStarted","Data":"7d0a17a227941d78eb28b0d35dea09ec29219d73000d5090766d649d44b00df6"} Nov 03 17:30:01.907881 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.907805 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" event={"ID":"aa47da19-7ea1-4601-bfba-18eca9086932","Type":"ContainerStarted","Data":"f5f5a0b39ac61e3fcc72533f37a3b852201e8bc36eabb51937322f626c4b8c6b"} Nov 03 17:30:01.936849 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.936802 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" podStartSLOduration=1.9367878539999999 podStartE2EDuration="1.936787854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:01.935548958 +0000 UTC m=+4545.524548927" watchObservedRunningTime="2026-11-03 17:30:01.936787854 +0000 UTC m=+4545.525787820" Nov 03 17:30:01.980536 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:01.980437 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" podStartSLOduration=1.9804229439999999 podStartE2EDuration="1.980422944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:01.977826995 +0000 UTC m=+4545.566826964" watchObservedRunningTime="2026-11-03 17:30:01.980422944 +0000 UTC m=+4545.569422912" Nov 03 17:30:02.065393 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.065333 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" podStartSLOduration=2.065315789 podStartE2EDuration="2.065315789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:02.062819888 +0000 UTC m=+4545.651819856" watchObservedRunningTime="2026-11-03 17:30:02.065315789 +0000 UTC m=+4545.654315756" Nov 03 17:30:02.109502 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.109443 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" podStartSLOduration=2.109428297 podStartE2EDuration="2.109428297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:02.107425564 +0000 UTC m=+4545.696425534" watchObservedRunningTime="2026-11-03 17:30:02.109428297 +0000 UTC m=+4545.698428283" Nov 03 17:30:02.151113 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.151053 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" podStartSLOduration=2.151038488 podStartE2EDuration="2.151038488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:02.1500812 +0000 UTC m=+4545.739081168" watchObservedRunningTime="2026-11-03 17:30:02.151038488 +0000 UTC m=+4545.740038455" Nov 03 17:30:02.194749 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.194680 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" podStartSLOduration=2.194666205 podStartE2EDuration="2.194666205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:02.192792319 +0000 UTC m=+4545.781792293" watchObservedRunningTime="2026-11-03 17:30:02.194666205 +0000 UTC m=+4545.783666172" Nov 03 17:30:02.917426 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.917397 4947 generic.go:356] "Generic (PLEG): container finished" podID="9644e7c0-4d23-498e-8a40-e76349d2e468" containerID="b4751f6670fa2510e7c8651ed86157267fcd856f75164c243303f76a408fc4d2" exitCode=60 Nov 03 17:30:02.917564 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.917479 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" event={"ID":"9644e7c0-4d23-498e-8a40-e76349d2e468","Type":"ContainerDied","Data":"b4751f6670fa2510e7c8651ed86157267fcd856f75164c243303f76a408fc4d2"} Nov 03 17:30:02.917894 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.917871 4947 scope.go:122] "RemoveContainer" containerID="b4751f6670fa2510e7c8651ed86157267fcd856f75164c243303f76a408fc4d2" Nov 03 17:30:02.919295 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.919274 4947 generic.go:356] "Generic (PLEG): container finished" podID="bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7" containerID="400e51361cf378ddf3c35438b3b3f970c7c051f909423e8c83ff4060c6a9398f" exitCode=60 Nov 03 17:30:02.919403 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.919357 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" event={"ID":"bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7","Type":"ContainerDied","Data":"400e51361cf378ddf3c35438b3b3f970c7c051f909423e8c83ff4060c6a9398f"} Nov 03 17:30:02.919607 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.919591 4947 scope.go:122] "RemoveContainer" containerID="400e51361cf378ddf3c35438b3b3f970c7c051f909423e8c83ff4060c6a9398f" Nov 03 17:30:02.921375 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.921356 4947 generic.go:356] "Generic (PLEG): container finished" podID="5af508d5-80f2-4f5a-b960-bbc2d604e97f" containerID="67bcba1ae1deb1161971408d6e77114efe3ba1cc04e198d31ede595c052e108a" exitCode=60 Nov 03 17:30:02.921505 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.921491 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" event={"ID":"5af508d5-80f2-4f5a-b960-bbc2d604e97f","Type":"ContainerDied","Data":"67bcba1ae1deb1161971408d6e77114efe3ba1cc04e198d31ede595c052e108a"} Nov 03 17:30:02.921707 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.921678 4947 scope.go:122] "RemoveContainer" containerID="67bcba1ae1deb1161971408d6e77114efe3ba1cc04e198d31ede595c052e108a" Nov 03 17:30:02.924071 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.924007 4947 generic.go:356] "Generic (PLEG): container finished" podID="01cb2467-7a13-4f96-9c67-2cc5104d99b9" containerID="21a05219585d2de3447639077ba863bd7648923b8b4f1d9ee0973efbb36c0690" exitCode=60 Nov 03 17:30:02.924168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.924140 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" event={"ID":"01cb2467-7a13-4f96-9c67-2cc5104d99b9","Type":"ContainerDied","Data":"21a05219585d2de3447639077ba863bd7648923b8b4f1d9ee0973efbb36c0690"} Nov 03 17:30:02.924212 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.924181 4947 scope.go:122] "RemoveContainer" containerID="fc7391cf191cc0e0bdc4bb3ce28302cacf106f3a6d50d568c42a427b485ae520" Nov 03 17:30:02.924309 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.924287 4947 scope.go:122] "RemoveContainer" containerID="21a05219585d2de3447639077ba863bd7648923b8b4f1d9ee0973efbb36c0690" Nov 03 17:30:02.924538 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:02.924514 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv_odh-ai-gateway-infra(01cb2467-7a13-4f96-9c67-2cc5104d99b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" podUID="01cb2467-7a13-4f96-9c67-2cc5104d99b9" Nov 03 17:30:02.926151 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.926129 4947 generic.go:356] "Generic (PLEG): container finished" podID="53960065-d46f-4926-82c0-6b180e9eab56" containerID="4dd1d2db6a81f22a859aee5739d122980a3a9c8127cb50f442d91cdb610ff2ee" exitCode=60 Nov 03 17:30:02.926250 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.926192 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" event={"ID":"53960065-d46f-4926-82c0-6b180e9eab56","Type":"ContainerDied","Data":"4dd1d2db6a81f22a859aee5739d122980a3a9c8127cb50f442d91cdb610ff2ee"} Nov 03 17:30:02.926360 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.926346 4947 scope.go:122] "RemoveContainer" containerID="4dd1d2db6a81f22a859aee5739d122980a3a9c8127cb50f442d91cdb610ff2ee" Nov 03 17:30:02.927646 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.927621 4947 generic.go:356] "Generic (PLEG): container finished" podID="0c92bab9-7a09-4cc8-b4a9-51bd85bfd818" containerID="79907f02daeb2b15be8cc2e499a309b7d8eb7a34d87bf9a2c32980fbbcf59a58" exitCode=60 Nov 03 17:30:02.927778 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.927706 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" event={"ID":"0c92bab9-7a09-4cc8-b4a9-51bd85bfd818","Type":"ContainerDied","Data":"79907f02daeb2b15be8cc2e499a309b7d8eb7a34d87bf9a2c32980fbbcf59a58"} Nov 03 17:30:02.927921 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.927904 4947 scope.go:122] "RemoveContainer" containerID="79907f02daeb2b15be8cc2e499a309b7d8eb7a34d87bf9a2c32980fbbcf59a58" Nov 03 17:30:02.930626 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.930606 4947 generic.go:356] "Generic (PLEG): container finished" podID="aa47da19-7ea1-4601-bfba-18eca9086932" containerID="7d0a17a227941d78eb28b0d35dea09ec29219d73000d5090766d649d44b00df6" exitCode=60 Nov 03 17:30:02.930763 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.930637 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" event={"ID":"aa47da19-7ea1-4601-bfba-18eca9086932","Type":"ContainerDied","Data":"7d0a17a227941d78eb28b0d35dea09ec29219d73000d5090766d649d44b00df6"} Nov 03 17:30:02.931075 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:02.930982 4947 scope.go:122] "RemoveContainer" containerID="7d0a17a227941d78eb28b0d35dea09ec29219d73000d5090766d649d44b00df6" Nov 03 17:30:03.939655 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:03.939624 4947 generic.go:356] "Generic (PLEG): container finished" podID="0c92bab9-7a09-4cc8-b4a9-51bd85bfd818" containerID="5ed58db758a693eb6de398058d1625ed7f4ef21dcc4106d94d2598b531ef497f" exitCode=60 Nov 03 17:30:03.940119 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:03.939722 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" event={"ID":"0c92bab9-7a09-4cc8-b4a9-51bd85bfd818","Type":"ContainerDied","Data":"5ed58db758a693eb6de398058d1625ed7f4ef21dcc4106d94d2598b531ef497f"} Nov 03 17:30:03.940119 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:03.939788 4947 scope.go:122] "RemoveContainer" containerID="79907f02daeb2b15be8cc2e499a309b7d8eb7a34d87bf9a2c32980fbbcf59a58" Nov 03 17:30:03.940119 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:03.940020 4947 scope.go:122] "RemoveContainer" containerID="5ed58db758a693eb6de398058d1625ed7f4ef21dcc4106d94d2598b531ef497f" Nov 03 17:30:03.940428 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:03.940399 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn_odh-ai-gateway-infra(0c92bab9-7a09-4cc8-b4a9-51bd85bfd818)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" podUID="0c92bab9-7a09-4cc8-b4a9-51bd85bfd818" Nov 03 17:30:03.941829 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:03.941807 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" event={"ID":"aa47da19-7ea1-4601-bfba-18eca9086932","Type":"ContainerStarted","Data":"af272d98348ca6c0aca26a62cbb5d66cf59978c2947c8feadd632f114d3e6368"} Nov 03 17:30:03.943567 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:03.943543 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" event={"ID":"9644e7c0-4d23-498e-8a40-e76349d2e468","Type":"ContainerStarted","Data":"0a9aeddfa6d2166e7b666650674ac9e84a85635621a6d578dbd6acf096030e85"} Nov 03 17:30:03.945858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:03.945835 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" event={"ID":"bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7","Type":"ContainerStarted","Data":"dbf2de76407f6c1bc91a361c458e2f1b47bd0a5d89b7677720bbfc00225bc40a"} Nov 03 17:30:03.947874 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:03.947852 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" event={"ID":"5af508d5-80f2-4f5a-b960-bbc2d604e97f","Type":"ContainerStarted","Data":"08d0cb50a7d710e4d97bc6977842f17575c9acca49253f82e05c38fcc63d2d19"} Nov 03 17:30:03.949516 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:03.949499 4947 scope.go:122] "RemoveContainer" containerID="21a05219585d2de3447639077ba863bd7648923b8b4f1d9ee0973efbb36c0690" Nov 03 17:30:03.949777 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:03.949757 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv_odh-ai-gateway-infra(01cb2467-7a13-4f96-9c67-2cc5104d99b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" podUID="01cb2467-7a13-4f96-9c67-2cc5104d99b9" Nov 03 17:30:03.950969 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:03.950948 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" event={"ID":"53960065-d46f-4926-82c0-6b180e9eab56","Type":"ContainerStarted","Data":"2df4ad5967f3679b532ac8a53cb47c39fe2ac2461f4152575e36746cae35d583"} Nov 03 17:30:04.960150 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.960111 4947 generic.go:356] "Generic (PLEG): container finished" podID="53960065-d46f-4926-82c0-6b180e9eab56" containerID="2df4ad5967f3679b532ac8a53cb47c39fe2ac2461f4152575e36746cae35d583" exitCode=60 Nov 03 17:30:04.960627 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.960182 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" event={"ID":"53960065-d46f-4926-82c0-6b180e9eab56","Type":"ContainerDied","Data":"2df4ad5967f3679b532ac8a53cb47c39fe2ac2461f4152575e36746cae35d583"} Nov 03 17:30:04.960627 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.960248 4947 scope.go:122] "RemoveContainer" containerID="4dd1d2db6a81f22a859aee5739d122980a3a9c8127cb50f442d91cdb610ff2ee" Nov 03 17:30:04.960627 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.960536 4947 scope.go:122] "RemoveContainer" containerID="2df4ad5967f3679b532ac8a53cb47c39fe2ac2461f4152575e36746cae35d583" Nov 03 17:30:04.960940 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:04.960909 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc_odh-ai-gateway-infra(53960065-d46f-4926-82c0-6b180e9eab56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" podUID="53960065-d46f-4926-82c0-6b180e9eab56" Nov 03 17:30:04.963664 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.963640 4947 scope.go:122] "RemoveContainer" containerID="5ed58db758a693eb6de398058d1625ed7f4ef21dcc4106d94d2598b531ef497f" Nov 03 17:30:04.963999 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:04.963977 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn_odh-ai-gateway-infra(0c92bab9-7a09-4cc8-b4a9-51bd85bfd818)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" podUID="0c92bab9-7a09-4cc8-b4a9-51bd85bfd818" Nov 03 17:30:04.965965 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.965940 4947 generic.go:356] "Generic (PLEG): container finished" podID="aa47da19-7ea1-4601-bfba-18eca9086932" containerID="af272d98348ca6c0aca26a62cbb5d66cf59978c2947c8feadd632f114d3e6368" exitCode=60 Nov 03 17:30:04.966129 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.966103 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" event={"ID":"aa47da19-7ea1-4601-bfba-18eca9086932","Type":"ContainerDied","Data":"af272d98348ca6c0aca26a62cbb5d66cf59978c2947c8feadd632f114d3e6368"} Nov 03 17:30:04.966302 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.966237 4947 scope.go:122] "RemoveContainer" containerID="af272d98348ca6c0aca26a62cbb5d66cf59978c2947c8feadd632f114d3e6368" Nov 03 17:30:04.966545 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:04.966522 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7_odh-ai-gateway-infra(aa47da19-7ea1-4601-bfba-18eca9086932)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" podUID="aa47da19-7ea1-4601-bfba-18eca9086932" Nov 03 17:30:04.968458 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.968438 4947 generic.go:356] "Generic (PLEG): container finished" podID="9644e7c0-4d23-498e-8a40-e76349d2e468" containerID="0a9aeddfa6d2166e7b666650674ac9e84a85635621a6d578dbd6acf096030e85" exitCode=60 Nov 03 17:30:04.968582 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.968521 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" event={"ID":"9644e7c0-4d23-498e-8a40-e76349d2e468","Type":"ContainerDied","Data":"0a9aeddfa6d2166e7b666650674ac9e84a85635621a6d578dbd6acf096030e85"} Nov 03 17:30:04.968776 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.968758 4947 scope.go:122] "RemoveContainer" containerID="0a9aeddfa6d2166e7b666650674ac9e84a85635621a6d578dbd6acf096030e85" Nov 03 17:30:04.969085 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:04.969061 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n_odh-ai-gateway-infra(9644e7c0-4d23-498e-8a40-e76349d2e468)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" podUID="9644e7c0-4d23-498e-8a40-e76349d2e468" Nov 03 17:30:04.970923 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.970900 4947 generic.go:356] "Generic (PLEG): container finished" podID="bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7" containerID="dbf2de76407f6c1bc91a361c458e2f1b47bd0a5d89b7677720bbfc00225bc40a" exitCode=60 Nov 03 17:30:04.971037 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.970988 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" event={"ID":"bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7","Type":"ContainerDied","Data":"dbf2de76407f6c1bc91a361c458e2f1b47bd0a5d89b7677720bbfc00225bc40a"} Nov 03 17:30:04.971206 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.971188 4947 scope.go:122] "RemoveContainer" containerID="dbf2de76407f6c1bc91a361c458e2f1b47bd0a5d89b7677720bbfc00225bc40a" Nov 03 17:30:04.971764 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:04.971735 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt_odh-ai-gateway-infra(bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" podUID="bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7" Nov 03 17:30:04.973791 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.973768 4947 generic.go:356] "Generic (PLEG): container finished" podID="5af508d5-80f2-4f5a-b960-bbc2d604e97f" containerID="08d0cb50a7d710e4d97bc6977842f17575c9acca49253f82e05c38fcc63d2d19" exitCode=60 Nov 03 17:30:04.973896 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.973847 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" event={"ID":"5af508d5-80f2-4f5a-b960-bbc2d604e97f","Type":"ContainerDied","Data":"08d0cb50a7d710e4d97bc6977842f17575c9acca49253f82e05c38fcc63d2d19"} Nov 03 17:30:04.974022 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.974009 4947 scope.go:122] "RemoveContainer" containerID="08d0cb50a7d710e4d97bc6977842f17575c9acca49253f82e05c38fcc63d2d19" Nov 03 17:30:04.974229 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:04.974213 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6_odh-ai-gateway-infra(5af508d5-80f2-4f5a-b960-bbc2d604e97f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" podUID="5af508d5-80f2-4f5a-b960-bbc2d604e97f" Nov 03 17:30:04.977349 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.977331 4947 scope.go:122] "RemoveContainer" containerID="7d0a17a227941d78eb28b0d35dea09ec29219d73000d5090766d649d44b00df6" Nov 03 17:30:04.990627 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:04.990611 4947 scope.go:122] "RemoveContainer" containerID="b4751f6670fa2510e7c8651ed86157267fcd856f75164c243303f76a408fc4d2" Nov 03 17:30:05.004359 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:05.004342 4947 scope.go:122] "RemoveContainer" containerID="400e51361cf378ddf3c35438b3b3f970c7c051f909423e8c83ff4060c6a9398f" Nov 03 17:30:05.017998 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:05.017981 4947 scope.go:122] "RemoveContainer" containerID="67bcba1ae1deb1161971408d6e77114efe3ba1cc04e198d31ede595c052e108a" Nov 03 17:30:05.983269 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:05.983238 4947 scope.go:122] "RemoveContainer" containerID="0a9aeddfa6d2166e7b666650674ac9e84a85635621a6d578dbd6acf096030e85" Nov 03 17:30:05.983752 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:05.983598 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n_odh-ai-gateway-infra(9644e7c0-4d23-498e-8a40-e76349d2e468)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" podUID="9644e7c0-4d23-498e-8a40-e76349d2e468" Nov 03 17:30:05.984870 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:05.984850 4947 scope.go:122] "RemoveContainer" containerID="dbf2de76407f6c1bc91a361c458e2f1b47bd0a5d89b7677720bbfc00225bc40a" Nov 03 17:30:05.985168 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:05.985144 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt_odh-ai-gateway-infra(bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" podUID="bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7" Nov 03 17:30:05.986421 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:05.986403 4947 scope.go:122] "RemoveContainer" containerID="08d0cb50a7d710e4d97bc6977842f17575c9acca49253f82e05c38fcc63d2d19" Nov 03 17:30:05.986716 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:05.986677 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6_odh-ai-gateway-infra(5af508d5-80f2-4f5a-b960-bbc2d604e97f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" podUID="5af508d5-80f2-4f5a-b960-bbc2d604e97f" Nov 03 17:30:05.987942 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:05.987921 4947 scope.go:122] "RemoveContainer" containerID="2df4ad5967f3679b532ac8a53cb47c39fe2ac2461f4152575e36746cae35d583" Nov 03 17:30:05.988260 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:05.988233 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc_odh-ai-gateway-infra(53960065-d46f-4926-82c0-6b180e9eab56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" podUID="53960065-d46f-4926-82c0-6b180e9eab56" Nov 03 17:30:05.993180 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:05.993163 4947 scope.go:122] "RemoveContainer" containerID="af272d98348ca6c0aca26a62cbb5d66cf59978c2947c8feadd632f114d3e6368" Nov 03 17:30:05.993405 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:05.993387 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7_odh-ai-gateway-infra(aa47da19-7ea1-4601-bfba-18eca9086932)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" podUID="aa47da19-7ea1-4601-bfba-18eca9086932" Nov 03 17:30:14.558614 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:14.558580 4947 scope.go:122] "RemoveContainer" containerID="2df4ad5967f3679b532ac8a53cb47c39fe2ac2461f4152575e36746cae35d583" Nov 03 17:30:15.069400 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:15.069302 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" event={"ID":"53960065-d46f-4926-82c0-6b180e9eab56","Type":"ContainerStarted","Data":"23f721bd3f8be771c50be396885368a6014cfefa559831642a1db889569edd32"} Nov 03 17:30:15.569423 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:15.559565 4947 scope.go:122] "RemoveContainer" containerID="21a05219585d2de3447639077ba863bd7648923b8b4f1d9ee0973efbb36c0690" Nov 03 17:30:15.569423 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:15.560572 4947 scope.go:122] "RemoveContainer" containerID="0a9aeddfa6d2166e7b666650674ac9e84a85635621a6d578dbd6acf096030e85" Nov 03 17:30:16.078609 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:16.078535 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" event={"ID":"9644e7c0-4d23-498e-8a40-e76349d2e468","Type":"ContainerStarted","Data":"2932fe3c38c980b21e27103e5573d395e1ce45533e2cd82137936a878008be90"} Nov 03 17:30:16.080522 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:16.080497 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" event={"ID":"01cb2467-7a13-4f96-9c67-2cc5104d99b9","Type":"ContainerStarted","Data":"b0b82a5f1acd4c66485da9525c498c8267355a5624a8d5fadf5fdc4f7dd49690"} Nov 03 17:30:16.082403 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:16.082381 4947 generic.go:356] "Generic (PLEG): container finished" podID="53960065-d46f-4926-82c0-6b180e9eab56" containerID="23f721bd3f8be771c50be396885368a6014cfefa559831642a1db889569edd32" exitCode=60 Nov 03 17:30:16.082497 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:16.082455 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" event={"ID":"53960065-d46f-4926-82c0-6b180e9eab56","Type":"ContainerDied","Data":"23f721bd3f8be771c50be396885368a6014cfefa559831642a1db889569edd32"} Nov 03 17:30:16.082497 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:16.082495 4947 scope.go:122] "RemoveContainer" containerID="2df4ad5967f3679b532ac8a53cb47c39fe2ac2461f4152575e36746cae35d583" Nov 03 17:30:16.082841 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:16.082821 4947 scope.go:122] "RemoveContainer" containerID="23f721bd3f8be771c50be396885368a6014cfefa559831642a1db889569edd32" Nov 03 17:30:16.083158 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:16.083132 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc_odh-ai-gateway-infra(53960065-d46f-4926-82c0-6b180e9eab56)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" podUID="53960065-d46f-4926-82c0-6b180e9eab56" Nov 03 17:30:16.125051 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:16.125028 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc"] Nov 03 17:30:16.169455 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:16.169407 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" podStartSLOduration=16.169394033 podStartE2EDuration="16.169394033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:30:16.166439787 +0000 UTC m=+4559.755439754" watchObservedRunningTime="2026-11-03 17:30:16.169394033 +0000 UTC m=+4559.758394001" Nov 03 17:30:16.562222 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:16.562194 4947 scope.go:122] "RemoveContainer" containerID="af272d98348ca6c0aca26a62cbb5d66cf59978c2947c8feadd632f114d3e6368" Nov 03 17:30:16.562395 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:16.562368 4947 scope.go:122] "RemoveContainer" containerID="08d0cb50a7d710e4d97bc6977842f17575c9acca49253f82e05c38fcc63d2d19" Nov 03 17:30:17.092113 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.092031 4947 generic.go:356] "Generic (PLEG): container finished" podID="9644e7c0-4d23-498e-8a40-e76349d2e468" containerID="2932fe3c38c980b21e27103e5573d395e1ce45533e2cd82137936a878008be90" exitCode=60 Nov 03 17:30:17.092113 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.092070 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" event={"ID":"9644e7c0-4d23-498e-8a40-e76349d2e468","Type":"ContainerDied","Data":"2932fe3c38c980b21e27103e5573d395e1ce45533e2cd82137936a878008be90"} Nov 03 17:30:17.092585 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.092164 4947 scope.go:122] "RemoveContainer" containerID="0a9aeddfa6d2166e7b666650674ac9e84a85635621a6d578dbd6acf096030e85" Nov 03 17:30:17.092585 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.092581 4947 scope.go:122] "RemoveContainer" containerID="2932fe3c38c980b21e27103e5573d395e1ce45533e2cd82137936a878008be90" Nov 03 17:30:17.093078 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:17.093037 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n_odh-ai-gateway-infra(9644e7c0-4d23-498e-8a40-e76349d2e468)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" podUID="9644e7c0-4d23-498e-8a40-e76349d2e468" Nov 03 17:30:17.094605 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.094574 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" event={"ID":"5af508d5-80f2-4f5a-b960-bbc2d604e97f","Type":"ContainerStarted","Data":"81a90af2bb4a878f164381fbf693ff929f37fc36318c8654c4b2086d2fbf6882"} Nov 03 17:30:17.096461 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.096432 4947 generic.go:356] "Generic (PLEG): container finished" podID="01cb2467-7a13-4f96-9c67-2cc5104d99b9" containerID="b0b82a5f1acd4c66485da9525c498c8267355a5624a8d5fadf5fdc4f7dd49690" exitCode=60 Nov 03 17:30:17.096576 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.096509 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" event={"ID":"01cb2467-7a13-4f96-9c67-2cc5104d99b9","Type":"ContainerDied","Data":"b0b82a5f1acd4c66485da9525c498c8267355a5624a8d5fadf5fdc4f7dd49690"} Nov 03 17:30:17.096957 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.096938 4947 scope.go:122] "RemoveContainer" containerID="b0b82a5f1acd4c66485da9525c498c8267355a5624a8d5fadf5fdc4f7dd49690" Nov 03 17:30:17.097242 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:17.097217 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv_odh-ai-gateway-infra(01cb2467-7a13-4f96-9c67-2cc5104d99b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" podUID="01cb2467-7a13-4f96-9c67-2cc5104d99b9" Nov 03 17:30:17.101470 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.100888 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" event={"ID":"aa47da19-7ea1-4601-bfba-18eca9086932","Type":"ContainerStarted","Data":"0e7b5999db1f4706f3f3962ea7c30d04bd388fd1bd5a87922a6fee2f272e9c12"} Nov 03 17:30:17.106591 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.106565 4947 scope.go:122] "RemoveContainer" containerID="21a05219585d2de3447639077ba863bd7648923b8b4f1d9ee0973efbb36c0690" Nov 03 17:30:17.155093 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.155062 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n"] Nov 03 17:30:17.248161 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.248123 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv"] Nov 03 17:30:17.311852 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.311824 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" Nov 03 17:30:17.342399 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.342318 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53960065-d46f-4926-82c0-6b180e9eab56-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53960065-d46f-4926-82c0-6b180e9eab56-maas-api-service-ca\") pod \"53960065-d46f-4926-82c0-6b180e9eab56\" (UID: \"53960065-d46f-4926-82c0-6b180e9eab56\") " Nov 03 17:30:17.342569 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.342407 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53960065-d46f-4926-82c0-6b180e9eab56-kube-api-access-kmkrd\" (UniqueName: \"kubernetes.io/projected/53960065-d46f-4926-82c0-6b180e9eab56-kube-api-access-kmkrd\") pod \"53960065-d46f-4926-82c0-6b180e9eab56\" (UID: \"53960065-d46f-4926-82c0-6b180e9eab56\") " Nov 03 17:30:17.342644 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.342621 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53960065-d46f-4926-82c0-6b180e9eab56-maas-api-service-ca" pod "53960065-d46f-4926-82c0-6b180e9eab56" (UID: "53960065-d46f-4926-82c0-6b180e9eab56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:17.343023 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.343002 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53960065-d46f-4926-82c0-6b180e9eab56-maas-api-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:17.344868 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.344838 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53960065-d46f-4926-82c0-6b180e9eab56-kube-api-access-kmkrd" pod "53960065-d46f-4926-82c0-6b180e9eab56" (UID: "53960065-d46f-4926-82c0-6b180e9eab56"). InnerVolumeSpecName "kube-api-access-kmkrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:17.444521 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.444493 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmkrd\" (UniqueName: \"kubernetes.io/projected/53960065-d46f-4926-82c0-6b180e9eab56-kube-api-access-kmkrd\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:17.558275 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:17.558241 4947 scope.go:122] "RemoveContainer" containerID="dbf2de76407f6c1bc91a361c458e2f1b47bd0a5d89b7677720bbfc00225bc40a" Nov 03 17:30:18.113032 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.113005 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" Nov 03 17:30:18.113441 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.113014 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc" event={"ID":"53960065-d46f-4926-82c0-6b180e9eab56","Type":"ContainerDied","Data":"e545ce802ace0c249c7b661a05f6e497cae5767887de4763f26c8404ea217001"} Nov 03 17:30:18.113441 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.113178 4947 scope.go:122] "RemoveContainer" containerID="23f721bd3f8be771c50be396885368a6014cfefa559831642a1db889569edd32" Nov 03 17:30:18.115582 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.115553 4947 generic.go:356] "Generic (PLEG): container finished" podID="aa47da19-7ea1-4601-bfba-18eca9086932" containerID="0e7b5999db1f4706f3f3962ea7c30d04bd388fd1bd5a87922a6fee2f272e9c12" exitCode=60 Nov 03 17:30:18.115737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.115646 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" event={"ID":"aa47da19-7ea1-4601-bfba-18eca9086932","Type":"ContainerDied","Data":"0e7b5999db1f4706f3f3962ea7c30d04bd388fd1bd5a87922a6fee2f272e9c12"} Nov 03 17:30:18.116173 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.116153 4947 scope.go:122] "RemoveContainer" containerID="0e7b5999db1f4706f3f3962ea7c30d04bd388fd1bd5a87922a6fee2f272e9c12" Nov 03 17:30:18.116557 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:18.116521 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7_odh-ai-gateway-infra(aa47da19-7ea1-4601-bfba-18eca9086932)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" podUID="aa47da19-7ea1-4601-bfba-18eca9086932" Nov 03 17:30:18.120769 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.120732 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" event={"ID":"bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7","Type":"ContainerStarted","Data":"85fba914f493b0aa938bc23acbaed796352437fd85d657ebf54fda983b655066"} Nov 03 17:30:18.123417 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.123391 4947 generic.go:356] "Generic (PLEG): container finished" podID="5af508d5-80f2-4f5a-b960-bbc2d604e97f" containerID="81a90af2bb4a878f164381fbf693ff929f37fc36318c8654c4b2086d2fbf6882" exitCode=60 Nov 03 17:30:18.123523 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.123427 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" event={"ID":"5af508d5-80f2-4f5a-b960-bbc2d604e97f","Type":"ContainerDied","Data":"81a90af2bb4a878f164381fbf693ff929f37fc36318c8654c4b2086d2fbf6882"} Nov 03 17:30:18.123839 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.123818 4947 scope.go:122] "RemoveContainer" containerID="81a90af2bb4a878f164381fbf693ff929f37fc36318c8654c4b2086d2fbf6882" Nov 03 17:30:18.124189 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:18.124149 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6_odh-ai-gateway-infra(5af508d5-80f2-4f5a-b960-bbc2d604e97f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" podUID="5af508d5-80f2-4f5a-b960-bbc2d604e97f" Nov 03 17:30:18.126578 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.126557 4947 scope.go:122] "RemoveContainer" containerID="af272d98348ca6c0aca26a62cbb5d66cf59978c2947c8feadd632f114d3e6368" Nov 03 17:30:18.148322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.147992 4947 scope.go:122] "RemoveContainer" containerID="08d0cb50a7d710e4d97bc6977842f17575c9acca49253f82e05c38fcc63d2d19" Nov 03 17:30:18.292751 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.289628 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc"] Nov 03 17:30:18.294757 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.294728 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e0e63d-29895450-jm9qc"] Nov 03 17:30:18.311051 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.311027 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" Nov 03 17:30:18.323512 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.323485 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7"] Nov 03 17:30:18.345093 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.345068 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" Nov 03 17:30:18.357991 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.357962 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9644e7c0-4d23-498e-8a40-e76349d2e468-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9644e7c0-4d23-498e-8a40-e76349d2e468-maas-api-service-ca\") pod \"9644e7c0-4d23-498e-8a40-e76349d2e468\" (UID: \"9644e7c0-4d23-498e-8a40-e76349d2e468\") " Nov 03 17:30:18.358108 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.358065 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9644e7c0-4d23-498e-8a40-e76349d2e468-kube-api-access-pxvtk\" (UniqueName: \"kubernetes.io/projected/9644e7c0-4d23-498e-8a40-e76349d2e468-kube-api-access-pxvtk\") pod \"9644e7c0-4d23-498e-8a40-e76349d2e468\" (UID: \"9644e7c0-4d23-498e-8a40-e76349d2e468\") " Nov 03 17:30:18.358156 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.358124 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01cb2467-7a13-4f96-9c67-2cc5104d99b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01cb2467-7a13-4f96-9c67-2cc5104d99b9-maas-api-service-ca\") pod \"01cb2467-7a13-4f96-9c67-2cc5104d99b9\" (UID: \"01cb2467-7a13-4f96-9c67-2cc5104d99b9\") " Nov 03 17:30:18.358190 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.358158 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01cb2467-7a13-4f96-9c67-2cc5104d99b9-kube-api-access-wscd5\" (UniqueName: \"kubernetes.io/projected/01cb2467-7a13-4f96-9c67-2cc5104d99b9-kube-api-access-wscd5\") pod \"01cb2467-7a13-4f96-9c67-2cc5104d99b9\" (UID: \"01cb2467-7a13-4f96-9c67-2cc5104d99b9\") " Nov 03 17:30:18.358389 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.358361 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9644e7c0-4d23-498e-8a40-e76349d2e468-maas-api-service-ca" pod "9644e7c0-4d23-498e-8a40-e76349d2e468" (UID: "9644e7c0-4d23-498e-8a40-e76349d2e468"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:18.358460 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.358440 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01cb2467-7a13-4f96-9c67-2cc5104d99b9-maas-api-service-ca" pod "01cb2467-7a13-4f96-9c67-2cc5104d99b9" (UID: "01cb2467-7a13-4f96-9c67-2cc5104d99b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:18.358953 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.358881 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/01cb2467-7a13-4f96-9c67-2cc5104d99b9-maas-api-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:18.358953 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.358908 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9644e7c0-4d23-498e-8a40-e76349d2e468-maas-api-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:18.360399 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.360374 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9644e7c0-4d23-498e-8a40-e76349d2e468-kube-api-access-pxvtk" pod "9644e7c0-4d23-498e-8a40-e76349d2e468" (UID: "9644e7c0-4d23-498e-8a40-e76349d2e468"). InnerVolumeSpecName "kube-api-access-pxvtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:18.360497 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.360398 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01cb2467-7a13-4f96-9c67-2cc5104d99b9-kube-api-access-wscd5" pod "01cb2467-7a13-4f96-9c67-2cc5104d99b9" (UID: "01cb2467-7a13-4f96-9c67-2cc5104d99b9"). InnerVolumeSpecName "kube-api-access-wscd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:18.420463 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.420385 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6"] Nov 03 17:30:18.460413 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.460375 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxvtk\" (UniqueName: \"kubernetes.io/projected/9644e7c0-4d23-498e-8a40-e76349d2e468-kube-api-access-pxvtk\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:18.460413 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.460412 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wscd5\" (UniqueName: \"kubernetes.io/projected/01cb2467-7a13-4f96-9c67-2cc5104d99b9-kube-api-access-wscd5\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:18.557918 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.557887 4947 scope.go:122] "RemoveContainer" containerID="5ed58db758a693eb6de398058d1625ed7f4ef21dcc4106d94d2598b531ef497f" Nov 03 17:30:18.567328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:18.567299 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53960065-d46f-4926-82c0-6b180e9eab56" path="/var/lib/kubelet/pods/53960065-d46f-4926-82c0-6b180e9eab56/volumes" Nov 03 17:30:19.134870 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.134843 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" Nov 03 17:30:19.135295 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.134845 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv" event={"ID":"01cb2467-7a13-4f96-9c67-2cc5104d99b9","Type":"ContainerDied","Data":"8d61e922f45464c6d1653f203f9573bb6a97be7a7ad76a23361f6a7466491805"} Nov 03 17:30:19.135295 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.135003 4947 scope.go:122] "RemoveContainer" containerID="b0b82a5f1acd4c66485da9525c498c8267355a5624a8d5fadf5fdc4f7dd49690" Nov 03 17:30:19.138341 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.138313 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" event={"ID":"0c92bab9-7a09-4cc8-b4a9-51bd85bfd818","Type":"ContainerStarted","Data":"a4b4e93f928d38e8c26085a861e7fcf02e149134c6b18e7b8864cce64e39bd69"} Nov 03 17:30:19.141670 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.141649 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" event={"ID":"9644e7c0-4d23-498e-8a40-e76349d2e468","Type":"ContainerDied","Data":"a5980bd56bc0f73693ee73f4eb070f2178407d8af050789f9e053e13e4266436"} Nov 03 17:30:19.141839 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.141760 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n" Nov 03 17:30:19.147121 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.144908 4947 generic.go:356] "Generic (PLEG): container finished" podID="bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7" containerID="85fba914f493b0aa938bc23acbaed796352437fd85d657ebf54fda983b655066" exitCode=60 Nov 03 17:30:19.147121 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.144972 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" event={"ID":"bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7","Type":"ContainerDied","Data":"85fba914f493b0aa938bc23acbaed796352437fd85d657ebf54fda983b655066"} Nov 03 17:30:19.147121 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.145386 4947 scope.go:122] "RemoveContainer" containerID="85fba914f493b0aa938bc23acbaed796352437fd85d657ebf54fda983b655066" Nov 03 17:30:19.147121 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:19.145757 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt_odh-ai-gateway-infra(bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" podUID="bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7" Nov 03 17:30:19.148477 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.148397 4947 scope.go:122] "RemoveContainer" containerID="2932fe3c38c980b21e27103e5573d395e1ce45533e2cd82137936a878008be90" Nov 03 17:30:19.159653 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.159633 4947 scope.go:122] "RemoveContainer" containerID="dbf2de76407f6c1bc91a361c458e2f1b47bd0a5d89b7677720bbfc00225bc40a" Nov 03 17:30:19.260230 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.260163 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt"] Nov 03 17:30:19.314379 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.314348 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv"] Nov 03 17:30:19.324459 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.324429 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-3a0e27-29895450-ckzwv"] Nov 03 17:30:19.333542 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.333516 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" Nov 03 17:30:19.371949 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.371922 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" Nov 03 17:30:19.373048 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.373029 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5af508d5-80f2-4f5a-b960-bbc2d604e97f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af508d5-80f2-4f5a-b960-bbc2d604e97f-maas-api-service-ca\") pod \"5af508d5-80f2-4f5a-b960-bbc2d604e97f\" (UID: \"5af508d5-80f2-4f5a-b960-bbc2d604e97f\") " Nov 03 17:30:19.373306 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.373293 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af508d5-80f2-4f5a-b960-bbc2d604e97f-kube-api-access-w7lf4\" (UniqueName: \"kubernetes.io/projected/5af508d5-80f2-4f5a-b960-bbc2d604e97f-kube-api-access-w7lf4\") pod \"5af508d5-80f2-4f5a-b960-bbc2d604e97f\" (UID: \"5af508d5-80f2-4f5a-b960-bbc2d604e97f\") " Nov 03 17:30:19.373388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.373366 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af508d5-80f2-4f5a-b960-bbc2d604e97f-maas-api-service-ca" pod "5af508d5-80f2-4f5a-b960-bbc2d604e97f" (UID: "5af508d5-80f2-4f5a-b960-bbc2d604e97f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:19.373677 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.373663 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5af508d5-80f2-4f5a-b960-bbc2d604e97f-maas-api-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:19.375514 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.375492 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af508d5-80f2-4f5a-b960-bbc2d604e97f-kube-api-access-w7lf4" pod "5af508d5-80f2-4f5a-b960-bbc2d604e97f" (UID: "5af508d5-80f2-4f5a-b960-bbc2d604e97f"). InnerVolumeSpecName "kube-api-access-w7lf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:19.400130 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.400105 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n"] Nov 03 17:30:19.406163 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.406140 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-491905-29895450-bzp5n"] Nov 03 17:30:19.475108 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.475070 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa47da19-7ea1-4601-bfba-18eca9086932-kube-api-access-l68hl\" (UniqueName: \"kubernetes.io/projected/aa47da19-7ea1-4601-bfba-18eca9086932-kube-api-access-l68hl\") pod \"aa47da19-7ea1-4601-bfba-18eca9086932\" (UID: \"aa47da19-7ea1-4601-bfba-18eca9086932\") " Nov 03 17:30:19.475262 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.475165 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa47da19-7ea1-4601-bfba-18eca9086932-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa47da19-7ea1-4601-bfba-18eca9086932-maas-api-service-ca\") pod \"aa47da19-7ea1-4601-bfba-18eca9086932\" (UID: \"aa47da19-7ea1-4601-bfba-18eca9086932\") " Nov 03 17:30:19.475604 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.475575 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa47da19-7ea1-4601-bfba-18eca9086932-maas-api-service-ca" pod "aa47da19-7ea1-4601-bfba-18eca9086932" (UID: "aa47da19-7ea1-4601-bfba-18eca9086932"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:19.476055 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.476029 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7lf4\" (UniqueName: \"kubernetes.io/projected/5af508d5-80f2-4f5a-b960-bbc2d604e97f-kube-api-access-w7lf4\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:19.476149 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.476061 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa47da19-7ea1-4601-bfba-18eca9086932-maas-api-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:19.477535 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.477505 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa47da19-7ea1-4601-bfba-18eca9086932-kube-api-access-l68hl" pod "aa47da19-7ea1-4601-bfba-18eca9086932" (UID: "aa47da19-7ea1-4601-bfba-18eca9086932"). InnerVolumeSpecName "kube-api-access-l68hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:19.577433 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:19.577387 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l68hl\" (UniqueName: \"kubernetes.io/projected/aa47da19-7ea1-4601-bfba-18eca9086932-kube-api-access-l68hl\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:20.152488 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.152455 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" Nov 03 17:30:20.152488 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.152481 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6" event={"ID":"5af508d5-80f2-4f5a-b960-bbc2d604e97f","Type":"ContainerDied","Data":"c776946c0e28099be55c54388afcbce46e0ec92de2415daea74d526c7be2fbb0"} Nov 03 17:30:20.152996 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.152542 4947 scope.go:122] "RemoveContainer" containerID="81a90af2bb4a878f164381fbf693ff929f37fc36318c8654c4b2086d2fbf6882" Nov 03 17:30:20.154681 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.154649 4947 generic.go:356] "Generic (PLEG): container finished" podID="0c92bab9-7a09-4cc8-b4a9-51bd85bfd818" containerID="a4b4e93f928d38e8c26085a861e7fcf02e149134c6b18e7b8864cce64e39bd69" exitCode=60 Nov 03 17:30:20.154823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.154741 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" event={"ID":"0c92bab9-7a09-4cc8-b4a9-51bd85bfd818","Type":"ContainerDied","Data":"a4b4e93f928d38e8c26085a861e7fcf02e149134c6b18e7b8864cce64e39bd69"} Nov 03 17:30:20.155212 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.155193 4947 scope.go:122] "RemoveContainer" containerID="a4b4e93f928d38e8c26085a861e7fcf02e149134c6b18e7b8864cce64e39bd69" Nov 03 17:30:20.155569 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:20.155546 4947 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn_odh-ai-gateway-infra(0c92bab9-7a09-4cc8-b4a9-51bd85bfd818)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" podUID="0c92bab9-7a09-4cc8-b4a9-51bd85bfd818" Nov 03 17:30:20.156065 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.156041 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" Nov 03 17:30:20.156158 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.156084 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7" event={"ID":"aa47da19-7ea1-4601-bfba-18eca9086932","Type":"ContainerDied","Data":"f5f5a0b39ac61e3fcc72533f37a3b852201e8bc36eabb51937322f626c4b8c6b"} Nov 03 17:30:20.172451 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.172427 4947 scope.go:122] "RemoveContainer" containerID="5ed58db758a693eb6de398058d1625ed7f4ef21dcc4106d94d2598b531ef497f" Nov 03 17:30:20.188082 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.188055 4947 scope.go:122] "RemoveContainer" containerID="0e7b5999db1f4706f3f3962ea7c30d04bd388fd1bd5a87922a6fee2f272e9c12" Nov 03 17:30:20.281932 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.281900 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn"] Nov 03 17:30:20.345391 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.345364 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7"] Nov 03 17:30:20.346528 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.346504 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" Nov 03 17:30:20.351375 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.351353 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c6f303-29895450-77rs7"] Nov 03 17:30:20.389180 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.389151 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7-kube-api-access-cqbwb\" (UniqueName: \"kubernetes.io/projected/bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7-kube-api-access-cqbwb\") pod \"bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7\" (UID: \"bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7\") " Nov 03 17:30:20.389352 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.389225 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7-maas-api-service-ca\") pod \"bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7\" (UID: \"bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7\") " Nov 03 17:30:20.390159 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.390131 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7-maas-api-service-ca" pod "bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7" (UID: "bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:20.390982 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.390957 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6"] Nov 03 17:30:20.391921 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.391893 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7-kube-api-access-cqbwb" pod "bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7" (UID: "bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7"). InnerVolumeSpecName "kube-api-access-cqbwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:20.394358 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.394339 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdbf0c-29895450-gmpb6"] Nov 03 17:30:20.514773 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.496749 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7-maas-api-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:20.514773 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.505471 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqbwb\" (UniqueName: \"kubernetes.io/projected/bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7-kube-api-access-cqbwb\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:20.588499 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.587164 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01cb2467-7a13-4f96-9c67-2cc5104d99b9" path="/var/lib/kubelet/pods/01cb2467-7a13-4f96-9c67-2cc5104d99b9/volumes" Nov 03 17:30:20.588499 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.587710 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af508d5-80f2-4f5a-b960-bbc2d604e97f" path="/var/lib/kubelet/pods/5af508d5-80f2-4f5a-b960-bbc2d604e97f/volumes" Nov 03 17:30:20.588499 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.588329 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9644e7c0-4d23-498e-8a40-e76349d2e468" path="/var/lib/kubelet/pods/9644e7c0-4d23-498e-8a40-e76349d2e468/volumes" Nov 03 17:30:20.589203 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:20.589170 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa47da19-7ea1-4601-bfba-18eca9086932" path="/var/lib/kubelet/pods/aa47da19-7ea1-4601-bfba-18eca9086932/volumes" Nov 03 17:30:21.174235 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:21.173833 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" event={"ID":"bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7","Type":"ContainerDied","Data":"52663dfa9f5ace73bcacb4ba99fa40e610af9dc87d3090400687ec67f130dbed"} Nov 03 17:30:21.174235 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:21.173909 4947 scope.go:122] "RemoveContainer" containerID="85fba914f493b0aa938bc23acbaed796352437fd85d657ebf54fda983b655066" Nov 03 17:30:21.174235 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:21.174008 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt" Nov 03 17:30:21.239126 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:21.239089 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt"] Nov 03 17:30:21.255151 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:21.255114 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-001802-29895450-d57tt"] Nov 03 17:30:21.348663 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:21.348636 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" Nov 03 17:30:21.422446 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:21.422408 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c92bab9-7a09-4cc8-b4a9-51bd85bfd818-kube-api-access-drvwk\" (UniqueName: \"kubernetes.io/projected/0c92bab9-7a09-4cc8-b4a9-51bd85bfd818-kube-api-access-drvwk\") pod \"0c92bab9-7a09-4cc8-b4a9-51bd85bfd818\" (UID: \"0c92bab9-7a09-4cc8-b4a9-51bd85bfd818\") " Nov 03 17:30:21.422639 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:21.422481 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c92bab9-7a09-4cc8-b4a9-51bd85bfd818-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c92bab9-7a09-4cc8-b4a9-51bd85bfd818-maas-api-service-ca\") pod \"0c92bab9-7a09-4cc8-b4a9-51bd85bfd818\" (UID: \"0c92bab9-7a09-4cc8-b4a9-51bd85bfd818\") " Nov 03 17:30:21.422898 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:21.422871 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c92bab9-7a09-4cc8-b4a9-51bd85bfd818-maas-api-service-ca" pod "0c92bab9-7a09-4cc8-b4a9-51bd85bfd818" (UID: "0c92bab9-7a09-4cc8-b4a9-51bd85bfd818"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:21.423316 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:21.423293 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c92bab9-7a09-4cc8-b4a9-51bd85bfd818-maas-api-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:21.424980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:21.424923 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c92bab9-7a09-4cc8-b4a9-51bd85bfd818-kube-api-access-drvwk" pod "0c92bab9-7a09-4cc8-b4a9-51bd85bfd818" (UID: "0c92bab9-7a09-4cc8-b4a9-51bd85bfd818"). InnerVolumeSpecName "kube-api-access-drvwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:21.524635 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:21.524596 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drvwk\" (UniqueName: \"kubernetes.io/projected/0c92bab9-7a09-4cc8-b4a9-51bd85bfd818-kube-api-access-drvwk\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:22.197307 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:22.197279 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" Nov 03 17:30:22.197841 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:22.197314 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn" event={"ID":"0c92bab9-7a09-4cc8-b4a9-51bd85bfd818","Type":"ContainerDied","Data":"ebd7cfb51fe861ffaed1f39d6aaf01cdf43c1917dc50fe520a35234110b1e24a"} Nov 03 17:30:22.197841 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:22.197532 4947 scope.go:122] "RemoveContainer" containerID="a4b4e93f928d38e8c26085a861e7fcf02e149134c6b18e7b8864cce64e39bd69" Nov 03 17:30:22.258671 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:22.258635 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn"] Nov 03 17:30:22.271669 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:22.271646 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-06fa9cd7-29895450-t7hzn"] Nov 03 17:30:22.566662 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:22.566630 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c92bab9-7a09-4cc8-b4a9-51bd85bfd818" path="/var/lib/kubelet/pods/0c92bab9-7a09-4cc8-b4a9-51bd85bfd818/volumes" Nov 03 17:30:22.567089 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:22.567074 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7" path="/var/lib/kubelet/pods/bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7/volumes" Nov 03 17:30:24.662412 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:24.662376 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:30:26.136722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:26.135915 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk"] Nov 03 17:30:26.136722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:26.136191 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" podUID="5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" containerName="istio-proxy" containerID="cri-o://d864ad35936a5425a0ad6caff3dd7b24ba87e7fc952a1787c0d75363f873e1ae" gracePeriod=30 Nov 03 17:30:27.038445 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:27.038410 4947 patch_prober.go:29] interesting pod/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:30:27.038612 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:27.038467 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" podUID="5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:30:30.131235 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:30.131198 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:30:31.538077 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.538052 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:30:31.628995 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.628909 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-credential-socket\") pod \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " Nov 03 17:30:31.628995 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.628966 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-kube-api-access-zfpmb\" (UniqueName: \"kubernetes.io/projected/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-kube-api-access-zfpmb\") pod \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " Nov 03 17:30:31.628995 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.628989 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-workload-certs\") pod \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " Nov 03 17:30:31.629322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.629034 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-envoy\") pod \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " Nov 03 17:30:31.629322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.629085 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-workload-socket\") pod \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " Nov 03 17:30:31.629322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.629139 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istiod-ca-cert\") pod \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " Nov 03 17:30:31.629322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.629226 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-credential-socket" pod "5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" (UID: "5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:31.629322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.629246 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-data\") pod \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " Nov 03 17:30:31.629574 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.629343 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-token\" (UniqueName: \"kubernetes.io/projected/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-token\") pod \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " Nov 03 17:30:31.629574 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.629410 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-podinfo\") pod \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\" (UID: \"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e\") " Nov 03 17:30:31.629574 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.629446 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-workload-socket" pod "5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" (UID: "5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:31.629574 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.629432 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-workload-certs" pod "5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" (UID: "5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:31.629986 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.629612 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-data" pod "5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" (UID: "5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:31.629986 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.629675 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istiod-ca-cert" pod "5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" (UID: "5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:31.630336 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.630309 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:31.630440 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.630342 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:31.630440 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.630358 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:31.630440 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.630372 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:31.630440 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.630385 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:31.631879 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.631850 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-token" pod "5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" (UID: "5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:31.632007 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.631895 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-kube-api-access-zfpmb" pod "5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" (UID: "5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e"). InnerVolumeSpecName "kube-api-access-zfpmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:31.632232 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.632197 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-podinfo" pod "5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" (UID: "5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:30:31.632336 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.632314 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-envoy" pod "5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" (UID: "5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:31.732475 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.732441 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:31.732475 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.732477 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:31.732680 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.732492 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfpmb\" (UniqueName: \"kubernetes.io/projected/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-kube-api-access-zfpmb\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:31.732680 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:31.732506 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:32.292471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:32.292436 4947 generic.go:356] "Generic (PLEG): container finished" podID="5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" containerID="d864ad35936a5425a0ad6caff3dd7b24ba87e7fc952a1787c0d75363f873e1ae" exitCode=0 Nov 03 17:30:32.292663 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:32.292485 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" Nov 03 17:30:32.292663 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:32.292488 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" event={"ID":"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e","Type":"ContainerDied","Data":"d864ad35936a5425a0ad6caff3dd7b24ba87e7fc952a1787c0d75363f873e1ae"} Nov 03 17:30:32.292663 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:32.292616 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk" event={"ID":"5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e","Type":"ContainerDied","Data":"c9b194aa96cb065a2e2409f2ee068f5a32af3957d7fb09bd367694afc64f100a"} Nov 03 17:30:32.292663 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:32.292647 4947 scope.go:122] "RemoveContainer" containerID="d864ad35936a5425a0ad6caff3dd7b24ba87e7fc952a1787c0d75363f873e1ae" Nov 03 17:30:32.303974 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:32.303953 4947 scope.go:122] "RemoveContainer" containerID="ddc9ab561808b41b10a6fe5eff8401fde77078b66c59bb58fd9579b47172a5f7" Nov 03 17:30:32.313315 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:32.313299 4947 scope.go:122] "RemoveContainer" containerID="d864ad35936a5425a0ad6caff3dd7b24ba87e7fc952a1787c0d75363f873e1ae" Nov 03 17:30:32.313577 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:32.313556 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d864ad35936a5425a0ad6caff3dd7b24ba87e7fc952a1787c0d75363f873e1ae\": container with ID starting with d864ad35936a5425a0ad6caff3dd7b24ba87e7fc952a1787c0d75363f873e1ae not found: ID does not exist" containerID="d864ad35936a5425a0ad6caff3dd7b24ba87e7fc952a1787c0d75363f873e1ae" Nov 03 17:30:32.313655 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:32.313587 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d864ad35936a5425a0ad6caff3dd7b24ba87e7fc952a1787c0d75363f873e1ae"} err="failed to get container status \"d864ad35936a5425a0ad6caff3dd7b24ba87e7fc952a1787c0d75363f873e1ae\": rpc error: code = NotFound desc = could not find container \"d864ad35936a5425a0ad6caff3dd7b24ba87e7fc952a1787c0d75363f873e1ae\": container with ID starting with d864ad35936a5425a0ad6caff3dd7b24ba87e7fc952a1787c0d75363f873e1ae not found: ID does not exist" Nov 03 17:30:32.313655 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:32.313610 4947 scope.go:122] "RemoveContainer" containerID="ddc9ab561808b41b10a6fe5eff8401fde77078b66c59bb58fd9579b47172a5f7" Nov 03 17:30:32.313912 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:32.313882 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddc9ab561808b41b10a6fe5eff8401fde77078b66c59bb58fd9579b47172a5f7\": container with ID starting with ddc9ab561808b41b10a6fe5eff8401fde77078b66c59bb58fd9579b47172a5f7 not found: ID does not exist" containerID="ddc9ab561808b41b10a6fe5eff8401fde77078b66c59bb58fd9579b47172a5f7" Nov 03 17:30:32.314001 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:32.313918 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddc9ab561808b41b10a6fe5eff8401fde77078b66c59bb58fd9579b47172a5f7"} err="failed to get container status \"ddc9ab561808b41b10a6fe5eff8401fde77078b66c59bb58fd9579b47172a5f7\": rpc error: code = NotFound desc = could not find container \"ddc9ab561808b41b10a6fe5eff8401fde77078b66c59bb58fd9579b47172a5f7\": container with ID starting with ddc9ab561808b41b10a6fe5eff8401fde77078b66c59bb58fd9579b47172a5f7 not found: ID does not exist" Nov 03 17:30:32.363746 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:32.363715 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk"] Nov 03 17:30:32.378050 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:32.378018 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdbf0c-openshift-default-579b8f46f-s2jfk"] Nov 03 17:30:32.545215 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:32.545136 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb"] Nov 03 17:30:32.566849 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:32.566811 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" path="/var/lib/kubelet/pods/5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e/volumes" Nov 03 17:30:32.579355 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:32.579330 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg"] Nov 03 17:30:37.545719 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:37.545637 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" podUID="e214eab6-0f39-415b-8ea9-8f583801475a" containerName="payload-processing" containerID="cri-o://e594230520f3e9fbacb1f2ed94c4e7b4c6ba4d906453aa8aa11a26065a3583d1" gracePeriod=25 Nov 03 17:30:37.580250 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:37.580206 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" podUID="cf0d4db9-e143-40e6-a677-2c2ea454ac3b" containerName="payload-pre-processing" containerID="cri-o://f319af464acdfa9275762f313958a8a49048f413a09ba804dddae9c02a3dcc24" gracePeriod=25 Nov 03 17:30:41.649426 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:41.649387 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" podUID="cf0d4db9-e143-40e6-a677-2c2ea454ac3b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:30:41.649860 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:41.649442 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" podUID="e214eab6-0f39-415b-8ea9-8f583801475a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:30:43.137830 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:43.137797 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x"] Nov 03 17:30:43.138244 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:43.138030 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" podUID="cab0d416-6e99-420f-9730-2e3b654e354f" containerName="maas-api" containerID="cri-o://dd91701ac9e15340b42fb8c021bac255e6b1754982131acc8e50ec27651a5cba" gracePeriod=30 Nov 03 17:30:43.231681 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:43.231650 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh"] Nov 03 17:30:43.401415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:43.401324 4947 generic.go:356] "Generic (PLEG): container finished" podID="cab0d416-6e99-420f-9730-2e3b654e354f" containerID="dd91701ac9e15340b42fb8c021bac255e6b1754982131acc8e50ec27651a5cba" exitCode=0 Nov 03 17:30:43.401415 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:43.401394 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" event={"ID":"cab0d416-6e99-420f-9730-2e3b654e354f","Type":"ContainerDied","Data":"dd91701ac9e15340b42fb8c021bac255e6b1754982131acc8e50ec27651a5cba"} Nov 03 17:30:43.439783 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:43.439759 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" Nov 03 17:30:43.640863 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:43.640826 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab0d416-6e99-420f-9730-2e3b654e354f-kube-api-access-7xrfg\" (UniqueName: \"kubernetes.io/projected/cab0d416-6e99-420f-9730-2e3b654e354f-kube-api-access-7xrfg\") pod \"cab0d416-6e99-420f-9730-2e3b654e354f\" (UID: \"cab0d416-6e99-420f-9730-2e3b654e354f\") " Nov 03 17:30:43.643389 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:43.643362 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cab0d416-6e99-420f-9730-2e3b654e354f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cab0d416-6e99-420f-9730-2e3b654e354f-metrics-certs\") pod \"cab0d416-6e99-420f-9730-2e3b654e354f\" (UID: \"cab0d416-6e99-420f-9730-2e3b654e354f\") " Nov 03 17:30:43.644518 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:43.644496 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cab0d416-6e99-420f-9730-2e3b654e354f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cab0d416-6e99-420f-9730-2e3b654e354f-maas-api-tls\") pod \"cab0d416-6e99-420f-9730-2e3b654e354f\" (UID: \"cab0d416-6e99-420f-9730-2e3b654e354f\") " Nov 03 17:30:43.647388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:43.647334 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab0d416-6e99-420f-9730-2e3b654e354f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab0d416-6e99-420f-9730-2e3b654e354f-openshift-service-ca\") pod \"cab0d416-6e99-420f-9730-2e3b654e354f\" (UID: \"cab0d416-6e99-420f-9730-2e3b654e354f\") " Nov 03 17:30:43.679733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:43.657196 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab0d416-6e99-420f-9730-2e3b654e354f-openshift-service-ca" pod "cab0d416-6e99-420f-9730-2e3b654e354f" (UID: "cab0d416-6e99-420f-9730-2e3b654e354f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:43.679733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:43.675857 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cab0d416-6e99-420f-9730-2e3b654e354f-metrics-certs" pod "cab0d416-6e99-420f-9730-2e3b654e354f" (UID: "cab0d416-6e99-420f-9730-2e3b654e354f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:30:43.679733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:43.678846 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab0d416-6e99-420f-9730-2e3b654e354f-kube-api-access-7xrfg" pod "cab0d416-6e99-420f-9730-2e3b654e354f" (UID: "cab0d416-6e99-420f-9730-2e3b654e354f"). InnerVolumeSpecName "kube-api-access-7xrfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:43.679733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:43.679459 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cab0d416-6e99-420f-9730-2e3b654e354f-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:43.679733 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:43.679486 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cab0d416-6e99-420f-9730-2e3b654e354f-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:43.681886 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:43.680150 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cab0d416-6e99-420f-9730-2e3b654e354f-maas-api-tls" pod "cab0d416-6e99-420f-9730-2e3b654e354f" (UID: "cab0d416-6e99-420f-9730-2e3b654e354f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:30:43.780675 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:43.780645 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xrfg\" (UniqueName: \"kubernetes.io/projected/cab0d416-6e99-420f-9730-2e3b654e354f-kube-api-access-7xrfg\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:43.780675 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:43.780673 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cab0d416-6e99-420f-9730-2e3b654e354f-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:44.409580 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:44.409497 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" Nov 03 17:30:44.409580 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:44.409530 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x" event={"ID":"cab0d416-6e99-420f-9730-2e3b654e354f","Type":"ContainerDied","Data":"7ac7a0e9e8f20480a580f070ecff5600123f23f981fe9666e4d9bda870cdd8c6"} Nov 03 17:30:44.410130 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:44.409602 4947 scope.go:122] "RemoveContainer" containerID="dd91701ac9e15340b42fb8c021bac255e6b1754982131acc8e50ec27651a5cba" Nov 03 17:30:44.467042 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:44.467020 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x"] Nov 03 17:30:44.479826 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:44.479804 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-9a80f0-59669497d4-5v99x"] Nov 03 17:30:44.566830 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:44.566805 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab0d416-6e99-420f-9730-2e3b654e354f" path="/var/lib/kubelet/pods/cab0d416-6e99-420f-9730-2e3b654e354f/volumes" Nov 03 17:30:47.629577 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:47.629545 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr"] Nov 03 17:30:47.630590 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:47.630554 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" podUID="f6e02969-c94d-47bf-9b80-278e570b6e0c" containerName="istio-proxy" containerID="cri-o://029eccfdf65187ac326b67918673a64969dff00bf9698b8c6fcecd4547305156" gracePeriod=30 Nov 03 17:30:48.232221 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:48.232180 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" podUID="97f5d9a3-9c9e-404f-a572-7821d58bcb8d" containerName="payload-processing" containerID="cri-o://023c332325aeab627e2aa1cb8ed66062893c8d9533457b02a8a53e6aabe13df5" gracePeriod=25 Nov 03 17:30:51.648794 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:51.648760 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" podUID="e214eab6-0f39-415b-8ea9-8f583801475a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:30:51.649197 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:51.648760 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" podUID="cf0d4db9-e143-40e6-a677-2c2ea454ac3b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:30:52.949610 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:52.949585 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 17:30:53.051338 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.051307 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-workload-certs\") pod \"f6e02969-c94d-47bf-9b80-278e570b6e0c\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " Nov 03 17:30:53.051515 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.051376 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-credential-socket\") pod \"f6e02969-c94d-47bf-9b80-278e570b6e0c\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " Nov 03 17:30:53.051515 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.051506 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-podinfo\") pod \"f6e02969-c94d-47bf-9b80-278e570b6e0c\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " Nov 03 17:30:53.051625 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.051578 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6e02969-c94d-47bf-9b80-278e570b6e0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6e02969-c94d-47bf-9b80-278e570b6e0c-istiod-ca-cert\") pod \"f6e02969-c94d-47bf-9b80-278e570b6e0c\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " Nov 03 17:30:53.051681 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.051625 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-workload-certs" pod "f6e02969-c94d-47bf-9b80-278e570b6e0c" (UID: "f6e02969-c94d-47bf-9b80-278e570b6e0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:53.051681 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.051649 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-envoy\") pod \"f6e02969-c94d-47bf-9b80-278e570b6e0c\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " Nov 03 17:30:53.051804 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.051752 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-data\") pod \"f6e02969-c94d-47bf-9b80-278e570b6e0c\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " Nov 03 17:30:53.051804 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.051776 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-credential-socket" pod "f6e02969-c94d-47bf-9b80-278e570b6e0c" (UID: "f6e02969-c94d-47bf-9b80-278e570b6e0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:53.051920 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.051899 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6e02969-c94d-47bf-9b80-278e570b6e0c-kube-api-access-cm62r\" (UniqueName: \"kubernetes.io/projected/f6e02969-c94d-47bf-9b80-278e570b6e0c-kube-api-access-cm62r\") pod \"f6e02969-c94d-47bf-9b80-278e570b6e0c\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " Nov 03 17:30:53.051975 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.051948 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-workload-socket\") pod \"f6e02969-c94d-47bf-9b80-278e570b6e0c\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " Nov 03 17:30:53.051975 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.051965 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-data" pod "f6e02969-c94d-47bf-9b80-278e570b6e0c" (UID: "f6e02969-c94d-47bf-9b80-278e570b6e0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:53.052084 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.051986 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-token\" (UniqueName: \"kubernetes.io/projected/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-token\") pod \"f6e02969-c94d-47bf-9b80-278e570b6e0c\" (UID: \"f6e02969-c94d-47bf-9b80-278e570b6e0c\") " Nov 03 17:30:53.052084 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.052028 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e02969-c94d-47bf-9b80-278e570b6e0c-istiod-ca-cert" pod "f6e02969-c94d-47bf-9b80-278e570b6e0c" (UID: "f6e02969-c94d-47bf-9b80-278e570b6e0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:53.052708 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.052269 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-workload-socket" pod "f6e02969-c94d-47bf-9b80-278e570b6e0c" (UID: "f6e02969-c94d-47bf-9b80-278e570b6e0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:53.053168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.053077 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:53.053168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.053102 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:53.053168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.053117 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:53.053168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.053132 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6e02969-c94d-47bf-9b80-278e570b6e0c-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:53.053168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.053146 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:53.054420 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.054392 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-envoy" pod "f6e02969-c94d-47bf-9b80-278e570b6e0c" (UID: "f6e02969-c94d-47bf-9b80-278e570b6e0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:53.054940 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.054905 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-podinfo" pod "f6e02969-c94d-47bf-9b80-278e570b6e0c" (UID: "f6e02969-c94d-47bf-9b80-278e570b6e0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:30:53.055022 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.054944 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e02969-c94d-47bf-9b80-278e570b6e0c-kube-api-access-cm62r" pod "f6e02969-c94d-47bf-9b80-278e570b6e0c" (UID: "f6e02969-c94d-47bf-9b80-278e570b6e0c"). InnerVolumeSpecName "kube-api-access-cm62r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:53.055022 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.055006 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-token" pod "f6e02969-c94d-47bf-9b80-278e570b6e0c" (UID: "f6e02969-c94d-47bf-9b80-278e570b6e0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:53.154320 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.154300 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm62r\" (UniqueName: \"kubernetes.io/projected/f6e02969-c94d-47bf-9b80-278e570b6e0c-kube-api-access-cm62r\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:53.154320 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.154322 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:53.154439 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.154331 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:53.154439 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.154340 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6e02969-c94d-47bf-9b80-278e570b6e0c-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:53.501471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.501431 4947 generic.go:356] "Generic (PLEG): container finished" podID="f6e02969-c94d-47bf-9b80-278e570b6e0c" containerID="029eccfdf65187ac326b67918673a64969dff00bf9698b8c6fcecd4547305156" exitCode=0 Nov 03 17:30:53.501660 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.501480 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" Nov 03 17:30:53.501660 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.501508 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" event={"ID":"f6e02969-c94d-47bf-9b80-278e570b6e0c","Type":"ContainerDied","Data":"029eccfdf65187ac326b67918673a64969dff00bf9698b8c6fcecd4547305156"} Nov 03 17:30:53.501660 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.501596 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr" event={"ID":"f6e02969-c94d-47bf-9b80-278e570b6e0c","Type":"ContainerDied","Data":"a2e23ed5a31f61bc54a6401338081f075a9b5e4be575eac28a2e360a0c9dd776"} Nov 03 17:30:53.501660 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.501624 4947 scope.go:122] "RemoveContainer" containerID="029eccfdf65187ac326b67918673a64969dff00bf9698b8c6fcecd4547305156" Nov 03 17:30:53.513268 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.513250 4947 scope.go:122] "RemoveContainer" containerID="d2ca55260de1c6e0baa5e5cbd26a937915311a9b0d56598e3a54ab7bb7df0622" Nov 03 17:30:53.522656 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.522640 4947 scope.go:122] "RemoveContainer" containerID="029eccfdf65187ac326b67918673a64969dff00bf9698b8c6fcecd4547305156" Nov 03 17:30:53.522938 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:53.522917 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"029eccfdf65187ac326b67918673a64969dff00bf9698b8c6fcecd4547305156\": container with ID starting with 029eccfdf65187ac326b67918673a64969dff00bf9698b8c6fcecd4547305156 not found: ID does not exist" containerID="029eccfdf65187ac326b67918673a64969dff00bf9698b8c6fcecd4547305156" Nov 03 17:30:53.523028 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.522951 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"029eccfdf65187ac326b67918673a64969dff00bf9698b8c6fcecd4547305156"} err="failed to get container status \"029eccfdf65187ac326b67918673a64969dff00bf9698b8c6fcecd4547305156\": rpc error: code = NotFound desc = could not find container \"029eccfdf65187ac326b67918673a64969dff00bf9698b8c6fcecd4547305156\": container with ID starting with 029eccfdf65187ac326b67918673a64969dff00bf9698b8c6fcecd4547305156 not found: ID does not exist" Nov 03 17:30:53.523028 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.522974 4947 scope.go:122] "RemoveContainer" containerID="d2ca55260de1c6e0baa5e5cbd26a937915311a9b0d56598e3a54ab7bb7df0622" Nov 03 17:30:53.523231 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:53.523215 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ca55260de1c6e0baa5e5cbd26a937915311a9b0d56598e3a54ab7bb7df0622\": container with ID starting with d2ca55260de1c6e0baa5e5cbd26a937915311a9b0d56598e3a54ab7bb7df0622 not found: ID does not exist" containerID="d2ca55260de1c6e0baa5e5cbd26a937915311a9b0d56598e3a54ab7bb7df0622" Nov 03 17:30:53.523271 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.523236 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ca55260de1c6e0baa5e5cbd26a937915311a9b0d56598e3a54ab7bb7df0622"} err="failed to get container status \"d2ca55260de1c6e0baa5e5cbd26a937915311a9b0d56598e3a54ab7bb7df0622\": rpc error: code = NotFound desc = could not find container \"d2ca55260de1c6e0baa5e5cbd26a937915311a9b0d56598e3a54ab7bb7df0622\": container with ID starting with d2ca55260de1c6e0baa5e5cbd26a937915311a9b0d56598e3a54ab7bb7df0622 not found: ID does not exist" Nov 03 17:30:53.558973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.558945 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr"] Nov 03 17:30:53.571843 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:53.571820 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-9a80f0-openshift-default-8497bd9c75-6nnqr"] Nov 03 17:30:54.264271 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:54.264243 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r"] Nov 03 17:30:54.264797 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:54.264427 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" podUID="d68c016e-91bf-4967-9d14-c6a0bea44dc4" containerName="istio-proxy" containerID="cri-o://4e1c0285516289527713c98153cfbd396f4e438b85cd4fe228cde0adeeafe9ac" gracePeriod=30 Nov 03 17:30:54.566495 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:54.566419 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6e02969-c94d-47bf-9b80-278e570b6e0c" path="/var/lib/kubelet/pods/f6e02969-c94d-47bf-9b80-278e570b6e0c/volumes" Nov 03 17:30:55.041804 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:55.041771 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" podUID="97f5d9a3-9c9e-404f-a572-7821d58bcb8d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:30:56.064726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:56.064628 4947 patch_prober.go:29] interesting pod/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:30:56.064726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:56.064688 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" podUID="d68c016e-91bf-4967-9d14-c6a0bea44dc4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:30:57.862340 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:57.862313 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" Nov 03 17:30:57.892827 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:57.892802 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" Nov 03 17:30:57.947223 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:57.947189 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e214eab6-0f39-415b-8ea9-8f583801475a-kube-api-access-8xf5p\" (UniqueName: \"kubernetes.io/projected/e214eab6-0f39-415b-8ea9-8f583801475a-kube-api-access-8xf5p\") pod \"e214eab6-0f39-415b-8ea9-8f583801475a\" (UID: \"e214eab6-0f39-415b-8ea9-8f583801475a\") " Nov 03 17:30:57.947430 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:57.947286 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf0d4db9-e143-40e6-a677-2c2ea454ac3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf0d4db9-e143-40e6-a677-2c2ea454ac3b-plugins-config-volume\") pod \"cf0d4db9-e143-40e6-a677-2c2ea454ac3b\" (UID: \"cf0d4db9-e143-40e6-a677-2c2ea454ac3b\") " Nov 03 17:30:57.947523 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:57.947496 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e214eab6-0f39-415b-8ea9-8f583801475a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e214eab6-0f39-415b-8ea9-8f583801475a-plugins-config-volume\") pod \"e214eab6-0f39-415b-8ea9-8f583801475a\" (UID: \"e214eab6-0f39-415b-8ea9-8f583801475a\") " Nov 03 17:30:57.947594 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:57.947567 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf0d4db9-e143-40e6-a677-2c2ea454ac3b-kube-api-access-9h2vp\" (UniqueName: \"kubernetes.io/projected/cf0d4db9-e143-40e6-a677-2c2ea454ac3b-kube-api-access-9h2vp\") pod \"cf0d4db9-e143-40e6-a677-2c2ea454ac3b\" (UID: \"cf0d4db9-e143-40e6-a677-2c2ea454ac3b\") " Nov 03 17:30:57.947837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:57.947681 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf0d4db9-e143-40e6-a677-2c2ea454ac3b-plugins-config-volume" pod "cf0d4db9-e143-40e6-a677-2c2ea454ac3b" (UID: "cf0d4db9-e143-40e6-a677-2c2ea454ac3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:57.947954 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:57.947886 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e214eab6-0f39-415b-8ea9-8f583801475a-plugins-config-volume" pod "e214eab6-0f39-415b-8ea9-8f583801475a" (UID: "e214eab6-0f39-415b-8ea9-8f583801475a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:57.948650 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:57.948615 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e214eab6-0f39-415b-8ea9-8f583801475a-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:57.948650 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:57.948652 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf0d4db9-e143-40e6-a677-2c2ea454ac3b-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:57.951123 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:57.951079 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e214eab6-0f39-415b-8ea9-8f583801475a-kube-api-access-8xf5p" pod "e214eab6-0f39-415b-8ea9-8f583801475a" (UID: "e214eab6-0f39-415b-8ea9-8f583801475a"). InnerVolumeSpecName "kube-api-access-8xf5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:57.951550 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:57.951372 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0d4db9-e143-40e6-a677-2c2ea454ac3b-kube-api-access-9h2vp" pod "cf0d4db9-e143-40e6-a677-2c2ea454ac3b" (UID: "cf0d4db9-e143-40e6-a677-2c2ea454ac3b"). InnerVolumeSpecName "kube-api-access-9h2vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:58.050369 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.050331 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h2vp\" (UniqueName: \"kubernetes.io/projected/cf0d4db9-e143-40e6-a677-2c2ea454ac3b-kube-api-access-9h2vp\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:58.050369 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.050364 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xf5p\" (UniqueName: \"kubernetes.io/projected/e214eab6-0f39-415b-8ea9-8f583801475a-kube-api-access-8xf5p\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:58.537765 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.537730 4947 generic.go:356] "Generic (PLEG): container finished" podID="cf0d4db9-e143-40e6-a677-2c2ea454ac3b" containerID="f319af464acdfa9275762f313958a8a49048f413a09ba804dddae9c02a3dcc24" exitCode=0 Nov 03 17:30:58.537765 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.537764 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" Nov 03 17:30:58.538074 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.537813 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" event={"ID":"cf0d4db9-e143-40e6-a677-2c2ea454ac3b","Type":"ContainerDied","Data":"f319af464acdfa9275762f313958a8a49048f413a09ba804dddae9c02a3dcc24"} Nov 03 17:30:58.538074 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.537901 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg" event={"ID":"cf0d4db9-e143-40e6-a677-2c2ea454ac3b","Type":"ContainerDied","Data":"379b06c0cac3c414f31b62b0c0ac92996f93f1b86667f006a2be5956d71b65df"} Nov 03 17:30:58.538074 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.537933 4947 scope.go:122] "RemoveContainer" containerID="f319af464acdfa9275762f313958a8a49048f413a09ba804dddae9c02a3dcc24" Nov 03 17:30:58.539418 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.539395 4947 generic.go:356] "Generic (PLEG): container finished" podID="e214eab6-0f39-415b-8ea9-8f583801475a" containerID="e594230520f3e9fbacb1f2ed94c4e7b4c6ba4d906453aa8aa11a26065a3583d1" exitCode=0 Nov 03 17:30:58.539538 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.539441 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" Nov 03 17:30:58.539538 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.539469 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" event={"ID":"e214eab6-0f39-415b-8ea9-8f583801475a","Type":"ContainerDied","Data":"e594230520f3e9fbacb1f2ed94c4e7b4c6ba4d906453aa8aa11a26065a3583d1"} Nov 03 17:30:58.539538 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.539503 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb" event={"ID":"e214eab6-0f39-415b-8ea9-8f583801475a","Type":"ContainerDied","Data":"21efee19b07a989f60e135013f1698f0852d8c1eed222cb500328242e784b097"} Nov 03 17:30:58.548711 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.548603 4947 scope.go:122] "RemoveContainer" containerID="f319af464acdfa9275762f313958a8a49048f413a09ba804dddae9c02a3dcc24" Nov 03 17:30:58.548995 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:58.548945 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f319af464acdfa9275762f313958a8a49048f413a09ba804dddae9c02a3dcc24\": container with ID starting with f319af464acdfa9275762f313958a8a49048f413a09ba804dddae9c02a3dcc24 not found: ID does not exist" containerID="f319af464acdfa9275762f313958a8a49048f413a09ba804dddae9c02a3dcc24" Nov 03 17:30:58.549079 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.548985 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f319af464acdfa9275762f313958a8a49048f413a09ba804dddae9c02a3dcc24"} err="failed to get container status \"f319af464acdfa9275762f313958a8a49048f413a09ba804dddae9c02a3dcc24\": rpc error: code = NotFound desc = could not find container \"f319af464acdfa9275762f313958a8a49048f413a09ba804dddae9c02a3dcc24\": container with ID starting with f319af464acdfa9275762f313958a8a49048f413a09ba804dddae9c02a3dcc24 not found: ID does not exist" Nov 03 17:30:58.549079 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.549008 4947 scope.go:122] "RemoveContainer" containerID="e594230520f3e9fbacb1f2ed94c4e7b4c6ba4d906453aa8aa11a26065a3583d1" Nov 03 17:30:58.572737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.572672 4947 scope.go:122] "RemoveContainer" containerID="e594230520f3e9fbacb1f2ed94c4e7b4c6ba4d906453aa8aa11a26065a3583d1" Nov 03 17:30:58.573018 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:30:58.572994 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e594230520f3e9fbacb1f2ed94c4e7b4c6ba4d906453aa8aa11a26065a3583d1\": container with ID starting with e594230520f3e9fbacb1f2ed94c4e7b4c6ba4d906453aa8aa11a26065a3583d1 not found: ID does not exist" containerID="e594230520f3e9fbacb1f2ed94c4e7b4c6ba4d906453aa8aa11a26065a3583d1" Nov 03 17:30:58.573114 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.573025 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e594230520f3e9fbacb1f2ed94c4e7b4c6ba4d906453aa8aa11a26065a3583d1"} err="failed to get container status \"e594230520f3e9fbacb1f2ed94c4e7b4c6ba4d906453aa8aa11a26065a3583d1\": rpc error: code = NotFound desc = could not find container \"e594230520f3e9fbacb1f2ed94c4e7b4c6ba4d906453aa8aa11a26065a3583d1\": container with ID starting with e594230520f3e9fbacb1f2ed94c4e7b4c6ba4d906453aa8aa11a26065a3583d1 not found: ID does not exist" Nov 03 17:30:58.594976 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.594953 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb"] Nov 03 17:30:58.599154 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.599133 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-9a80f0-5fbbcc57-z7nqb"] Nov 03 17:30:58.648595 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.648557 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg"] Nov 03 17:30:58.654296 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:58.654268 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-9a80f0-dbd6b55d7-4qgtg"] Nov 03 17:30:59.550664 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.550633 4947 generic.go:356] "Generic (PLEG): container finished" podID="d68c016e-91bf-4967-9d14-c6a0bea44dc4" containerID="4e1c0285516289527713c98153cfbd396f4e438b85cd4fe228cde0adeeafe9ac" exitCode=0 Nov 03 17:30:59.551050 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.550716 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" event={"ID":"d68c016e-91bf-4967-9d14-c6a0bea44dc4","Type":"ContainerDied","Data":"4e1c0285516289527713c98153cfbd396f4e438b85cd4fe228cde0adeeafe9ac"} Nov 03 17:30:59.551050 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.550768 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" event={"ID":"d68c016e-91bf-4967-9d14-c6a0bea44dc4","Type":"ContainerDied","Data":"b379dd5966b0ca3ab1a99296b9e9e0bfe338ae84b1e24d5347cd6dcbb838ccc8"} Nov 03 17:30:59.551050 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.550786 4947 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b379dd5966b0ca3ab1a99296b9e9e0bfe338ae84b1e24d5347cd6dcbb838ccc8" Nov 03 17:30:59.551169 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.551144 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 17:30:59.670112 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.670076 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-workload-certs\") pod \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " Nov 03 17:30:59.670285 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.670157 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-workload-socket\") pod \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " Nov 03 17:30:59.670285 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.670193 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-envoy\") pod \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " Nov 03 17:30:59.670444 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.670316 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istiod-ca-cert\") pod \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " Nov 03 17:30:59.670444 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.670347 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d68c016e-91bf-4967-9d14-c6a0bea44dc4-kube-api-access-7kjtx\" (UniqueName: \"kubernetes.io/projected/d68c016e-91bf-4967-9d14-c6a0bea44dc4-kube-api-access-7kjtx\") pod \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " Nov 03 17:30:59.670444 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.670372 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-token\" (UniqueName: \"kubernetes.io/projected/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-token\") pod \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " Nov 03 17:30:59.670444 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.670405 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-credential-socket\") pod \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " Nov 03 17:30:59.670444 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.670429 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-workload-certs" pod "d68c016e-91bf-4967-9d14-c6a0bea44dc4" (UID: "d68c016e-91bf-4967-9d14-c6a0bea44dc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:59.670444 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.670438 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-data\") pod \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " Nov 03 17:30:59.670841 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.670508 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-workload-socket" pod "d68c016e-91bf-4967-9d14-c6a0bea44dc4" (UID: "d68c016e-91bf-4967-9d14-c6a0bea44dc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:59.670841 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.670611 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-podinfo\") pod \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\" (UID: \"d68c016e-91bf-4967-9d14-c6a0bea44dc4\") " Nov 03 17:30:59.670841 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.670712 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-data" pod "d68c016e-91bf-4967-9d14-c6a0bea44dc4" (UID: "d68c016e-91bf-4967-9d14-c6a0bea44dc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:59.670841 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.670740 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-credential-socket" pod "d68c016e-91bf-4967-9d14-c6a0bea44dc4" (UID: "d68c016e-91bf-4967-9d14-c6a0bea44dc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:59.671050 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.670980 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istiod-ca-cert" pod "d68c016e-91bf-4967-9d14-c6a0bea44dc4" (UID: "d68c016e-91bf-4967-9d14-c6a0bea44dc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:30:59.672407 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.672012 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:59.672407 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.672039 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:59.672407 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.672053 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:59.672407 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.672068 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:59.672407 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.672082 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:59.673147 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.673121 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-podinfo" pod "d68c016e-91bf-4967-9d14-c6a0bea44dc4" (UID: "d68c016e-91bf-4967-9d14-c6a0bea44dc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:30:59.673324 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.673298 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-envoy" pod "d68c016e-91bf-4967-9d14-c6a0bea44dc4" (UID: "d68c016e-91bf-4967-9d14-c6a0bea44dc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:30:59.673428 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.673381 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-token" pod "d68c016e-91bf-4967-9d14-c6a0bea44dc4" (UID: "d68c016e-91bf-4967-9d14-c6a0bea44dc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:59.673428 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.673406 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68c016e-91bf-4967-9d14-c6a0bea44dc4-kube-api-access-7kjtx" pod "d68c016e-91bf-4967-9d14-c6a0bea44dc4" (UID: "d68c016e-91bf-4967-9d14-c6a0bea44dc4"). InnerVolumeSpecName "kube-api-access-7kjtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:30:59.773290 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.773254 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kjtx\" (UniqueName: \"kubernetes.io/projected/d68c016e-91bf-4967-9d14-c6a0bea44dc4-kube-api-access-7kjtx\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:59.773290 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.773287 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:59.773290 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.773297 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:30:59.773569 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:30:59.773305 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d68c016e-91bf-4967-9d14-c6a0bea44dc4-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:00.131173 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:31:00.131135 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:31:00.556948 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:00.556917 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r" Nov 03 17:31:00.566428 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:00.566403 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf0d4db9-e143-40e6-a677-2c2ea454ac3b" path="/var/lib/kubelet/pods/cf0d4db9-e143-40e6-a677-2c2ea454ac3b/volumes" Nov 03 17:31:00.566857 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:00.566843 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e214eab6-0f39-415b-8ea9-8f583801475a" path="/var/lib/kubelet/pods/e214eab6-0f39-415b-8ea9-8f583801475a/volumes" Nov 03 17:31:00.627001 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:00.626961 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r"] Nov 03 17:31:00.640139 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:00.640107 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3a2fa2-openshift-default-5459b779d9-rb92r"] Nov 03 17:31:02.566562 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:02.566522 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d68c016e-91bf-4967-9d14-c6a0bea44dc4" path="/var/lib/kubelet/pods/d68c016e-91bf-4967-9d14-c6a0bea44dc4/volumes" Nov 03 17:31:05.042180 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:05.042144 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" podUID="97f5d9a3-9c9e-404f-a572-7821d58bcb8d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:31:08.518041 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:08.518018 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" Nov 03 17:31:08.529190 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:08.529168 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f5d9a3-9c9e-404f-a572-7821d58bcb8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f5d9a3-9c9e-404f-a572-7821d58bcb8d-plugins-config-volume\") pod \"97f5d9a3-9c9e-404f-a572-7821d58bcb8d\" (UID: \"97f5d9a3-9c9e-404f-a572-7821d58bcb8d\") " Nov 03 17:31:08.529314 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:08.529241 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f5d9a3-9c9e-404f-a572-7821d58bcb8d-kube-api-access-nsfd5\" (UniqueName: \"kubernetes.io/projected/97f5d9a3-9c9e-404f-a572-7821d58bcb8d-kube-api-access-nsfd5\") pod \"97f5d9a3-9c9e-404f-a572-7821d58bcb8d\" (UID: \"97f5d9a3-9c9e-404f-a572-7821d58bcb8d\") " Nov 03 17:31:08.529517 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:08.529496 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f5d9a3-9c9e-404f-a572-7821d58bcb8d-plugins-config-volume" pod "97f5d9a3-9c9e-404f-a572-7821d58bcb8d" (UID: "97f5d9a3-9c9e-404f-a572-7821d58bcb8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:08.529851 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:08.529831 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f5d9a3-9c9e-404f-a572-7821d58bcb8d-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:08.531419 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:08.531396 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f5d9a3-9c9e-404f-a572-7821d58bcb8d-kube-api-access-nsfd5" pod "97f5d9a3-9c9e-404f-a572-7821d58bcb8d" (UID: "97f5d9a3-9c9e-404f-a572-7821d58bcb8d"). InnerVolumeSpecName "kube-api-access-nsfd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:08.623568 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:08.623533 4947 generic.go:356] "Generic (PLEG): container finished" podID="97f5d9a3-9c9e-404f-a572-7821d58bcb8d" containerID="023c332325aeab627e2aa1cb8ed66062893c8d9533457b02a8a53e6aabe13df5" exitCode=0 Nov 03 17:31:08.623782 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:08.623581 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" Nov 03 17:31:08.623782 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:08.623622 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" event={"ID":"97f5d9a3-9c9e-404f-a572-7821d58bcb8d","Type":"ContainerDied","Data":"023c332325aeab627e2aa1cb8ed66062893c8d9533457b02a8a53e6aabe13df5"} Nov 03 17:31:08.623782 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:08.623685 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh" event={"ID":"97f5d9a3-9c9e-404f-a572-7821d58bcb8d","Type":"ContainerDied","Data":"52fb7adc0376e2271ef291793319085a374e4f8bee29577d901c54ac93a66aa5"} Nov 03 17:31:08.623782 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:08.623746 4947 scope.go:122] "RemoveContainer" containerID="023c332325aeab627e2aa1cb8ed66062893c8d9533457b02a8a53e6aabe13df5" Nov 03 17:31:08.631175 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:08.631152 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsfd5\" (UniqueName: \"kubernetes.io/projected/97f5d9a3-9c9e-404f-a572-7821d58bcb8d-kube-api-access-nsfd5\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:08.634451 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:08.634434 4947 scope.go:122] "RemoveContainer" containerID="023c332325aeab627e2aa1cb8ed66062893c8d9533457b02a8a53e6aabe13df5" Nov 03 17:31:08.634685 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:31:08.634667 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"023c332325aeab627e2aa1cb8ed66062893c8d9533457b02a8a53e6aabe13df5\": container with ID starting with 023c332325aeab627e2aa1cb8ed66062893c8d9533457b02a8a53e6aabe13df5 not found: ID does not exist" containerID="023c332325aeab627e2aa1cb8ed66062893c8d9533457b02a8a53e6aabe13df5" Nov 03 17:31:08.634786 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:08.634707 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"023c332325aeab627e2aa1cb8ed66062893c8d9533457b02a8a53e6aabe13df5"} err="failed to get container status \"023c332325aeab627e2aa1cb8ed66062893c8d9533457b02a8a53e6aabe13df5\": rpc error: code = NotFound desc = could not find container \"023c332325aeab627e2aa1cb8ed66062893c8d9533457b02a8a53e6aabe13df5\": container with ID starting with 023c332325aeab627e2aa1cb8ed66062893c8d9533457b02a8a53e6aabe13df5 not found: ID does not exist" Nov 03 17:31:08.678327 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:08.678304 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh"] Nov 03 17:31:08.682358 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:08.682339 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6046c2-bdfd46647-bsnqh"] Nov 03 17:31:09.521789 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:09.521757 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68"] Nov 03 17:31:10.592579 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:10.592530 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f5d9a3-9c9e-404f-a572-7821d58bcb8d" path="/var/lib/kubelet/pods/97f5d9a3-9c9e-404f-a572-7821d58bcb8d/volumes" Nov 03 17:31:11.798486 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:11.798441 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794"] Nov 03 17:31:11.799036 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:11.798772 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" podUID="fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8" containerName="maas-api" containerID="cri-o://b885ff33d43edec6b26ba54fab5b4a8dbe20e94a2a674eb5ae4cdfa78d93b08c" gracePeriod=30 Nov 03 17:31:12.089491 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.089461 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" Nov 03 17:31:12.208721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.205875 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-kube-api-access-pljhm\" (UniqueName: \"kubernetes.io/projected/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-kube-api-access-pljhm\") pod \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\" (UID: \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\") " Nov 03 17:31:12.230727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.227924 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-openshift-service-ca\") pod \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\" (UID: \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\") " Nov 03 17:31:12.230727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.228197 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-metrics-certs\") pod \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\" (UID: \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\") " Nov 03 17:31:12.230727 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.228263 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-maas-api-tls\") pod \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\" (UID: \"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8\") " Nov 03 17:31:12.240976 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.240903 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-openshift-service-ca" pod "fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8" (UID: "fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:12.241355 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.241311 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-maas-api-tls" pod "fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8" (UID: "fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:31:12.242016 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.241965 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-kube-api-access-pljhm" pod "fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8" (UID: "fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8"). InnerVolumeSpecName "kube-api-access-pljhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:12.243331 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.243296 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-metrics-certs" pod "fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8" (UID: "fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:31:12.330285 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.330207 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:12.330285 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.330235 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:12.330285 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.330244 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pljhm\" (UniqueName: \"kubernetes.io/projected/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-kube-api-access-pljhm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:12.330285 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.330255 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:12.658603 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.658573 4947 generic.go:356] "Generic (PLEG): container finished" podID="fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8" containerID="b885ff33d43edec6b26ba54fab5b4a8dbe20e94a2a674eb5ae4cdfa78d93b08c" exitCode=0 Nov 03 17:31:12.658806 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.658620 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" Nov 03 17:31:12.658806 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.658666 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" event={"ID":"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8","Type":"ContainerDied","Data":"b885ff33d43edec6b26ba54fab5b4a8dbe20e94a2a674eb5ae4cdfa78d93b08c"} Nov 03 17:31:12.658806 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.658731 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794" event={"ID":"fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8","Type":"ContainerDied","Data":"4755a68dfe07e219ad1925631d40487bffc184b0d478d54b4d844229b0de33a2"} Nov 03 17:31:12.658806 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.658758 4947 scope.go:122] "RemoveContainer" containerID="b885ff33d43edec6b26ba54fab5b4a8dbe20e94a2a674eb5ae4cdfa78d93b08c" Nov 03 17:31:12.668474 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.668455 4947 scope.go:122] "RemoveContainer" containerID="b885ff33d43edec6b26ba54fab5b4a8dbe20e94a2a674eb5ae4cdfa78d93b08c" Nov 03 17:31:12.668839 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:31:12.668813 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b885ff33d43edec6b26ba54fab5b4a8dbe20e94a2a674eb5ae4cdfa78d93b08c\": container with ID starting with b885ff33d43edec6b26ba54fab5b4a8dbe20e94a2a674eb5ae4cdfa78d93b08c not found: ID does not exist" containerID="b885ff33d43edec6b26ba54fab5b4a8dbe20e94a2a674eb5ae4cdfa78d93b08c" Nov 03 17:31:12.668910 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.668851 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b885ff33d43edec6b26ba54fab5b4a8dbe20e94a2a674eb5ae4cdfa78d93b08c"} err="failed to get container status \"b885ff33d43edec6b26ba54fab5b4a8dbe20e94a2a674eb5ae4cdfa78d93b08c\": rpc error: code = NotFound desc = could not find container \"b885ff33d43edec6b26ba54fab5b4a8dbe20e94a2a674eb5ae4cdfa78d93b08c\": container with ID starting with b885ff33d43edec6b26ba54fab5b4a8dbe20e94a2a674eb5ae4cdfa78d93b08c not found: ID does not exist" Nov 03 17:31:12.714346 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.714318 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794"] Nov 03 17:31:12.727535 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:12.727507 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-06fa9cd7-5858887f46-km794"] Nov 03 17:31:14.117101 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:14.117067 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9"] Nov 03 17:31:14.117549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:14.117258 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" podUID="6a74f616-c592-4caa-9918-b774a56814af" containerName="istio-proxy" containerID="cri-o://f63d19213c2f35c65d339c67ecce2127f681b643d2a8bde4e99a509ed6e8d338" gracePeriod=30 Nov 03 17:31:14.566509 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:14.566479 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8" path="/var/lib/kubelet/pods/fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8/volumes" Nov 03 17:31:19.395433 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.395353 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:31:19.439956 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.439927 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a74f616-c592-4caa-9918-b774a56814af-istio-token\" (UniqueName: \"kubernetes.io/projected/6a74f616-c592-4caa-9918-b774a56814af-istio-token\") pod \"6a74f616-c592-4caa-9918-b774a56814af\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " Nov 03 17:31:19.440135 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.440005 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a74f616-c592-4caa-9918-b774a56814af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a74f616-c592-4caa-9918-b774a56814af-istio-podinfo\") pod \"6a74f616-c592-4caa-9918-b774a56814af\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " Nov 03 17:31:19.440135 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.440100 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-istio-data\") pod \"6a74f616-c592-4caa-9918-b774a56814af\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " Nov 03 17:31:19.440227 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.440158 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a74f616-c592-4caa-9918-b774a56814af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a74f616-c592-4caa-9918-b774a56814af-istiod-ca-cert\") pod \"6a74f616-c592-4caa-9918-b774a56814af\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " Nov 03 17:31:19.440227 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.440199 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-workload-socket\") pod \"6a74f616-c592-4caa-9918-b774a56814af\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " Nov 03 17:31:19.440337 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.440232 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-credential-socket\") pod \"6a74f616-c592-4caa-9918-b774a56814af\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " Nov 03 17:31:19.440337 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.440268 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-workload-certs\") pod \"6a74f616-c592-4caa-9918-b774a56814af\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " Nov 03 17:31:19.440464 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.440337 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-istio-data" pod "6a74f616-c592-4caa-9918-b774a56814af" (UID: "6a74f616-c592-4caa-9918-b774a56814af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:19.440464 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.440427 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-istio-envoy\") pod \"6a74f616-c592-4caa-9918-b774a56814af\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " Nov 03 17:31:19.440571 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.440492 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-workload-socket" pod "6a74f616-c592-4caa-9918-b774a56814af" (UID: "6a74f616-c592-4caa-9918-b774a56814af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:19.440571 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.440547 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a74f616-c592-4caa-9918-b774a56814af-kube-api-access-ptqq2\" (UniqueName: \"kubernetes.io/projected/6a74f616-c592-4caa-9918-b774a56814af-kube-api-access-ptqq2\") pod \"6a74f616-c592-4caa-9918-b774a56814af\" (UID: \"6a74f616-c592-4caa-9918-b774a56814af\") " Nov 03 17:31:19.440571 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.440552 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-credential-socket" pod "6a74f616-c592-4caa-9918-b774a56814af" (UID: "6a74f616-c592-4caa-9918-b774a56814af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:19.440741 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.440568 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-workload-certs" pod "6a74f616-c592-4caa-9918-b774a56814af" (UID: "6a74f616-c592-4caa-9918-b774a56814af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:19.440741 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.440587 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a74f616-c592-4caa-9918-b774a56814af-istiod-ca-cert" pod "6a74f616-c592-4caa-9918-b774a56814af" (UID: "6a74f616-c592-4caa-9918-b774a56814af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:19.441305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.441035 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:19.441305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.441055 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a74f616-c592-4caa-9918-b774a56814af-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:19.441305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.441069 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:19.441305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.441084 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:19.441305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.441102 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:19.442462 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.442431 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a74f616-c592-4caa-9918-b774a56814af-istio-token" pod "6a74f616-c592-4caa-9918-b774a56814af" (UID: "6a74f616-c592-4caa-9918-b774a56814af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:19.442588 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.442549 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a74f616-c592-4caa-9918-b774a56814af-istio-podinfo" pod "6a74f616-c592-4caa-9918-b774a56814af" (UID: "6a74f616-c592-4caa-9918-b774a56814af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:31:19.442812 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.442779 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-istio-envoy" pod "6a74f616-c592-4caa-9918-b774a56814af" (UID: "6a74f616-c592-4caa-9918-b774a56814af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:19.442902 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.442836 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a74f616-c592-4caa-9918-b774a56814af-kube-api-access-ptqq2" pod "6a74f616-c592-4caa-9918-b774a56814af" (UID: "6a74f616-c592-4caa-9918-b774a56814af"). InnerVolumeSpecName "kube-api-access-ptqq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:19.541975 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.541947 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptqq2\" (UniqueName: \"kubernetes.io/projected/6a74f616-c592-4caa-9918-b774a56814af-kube-api-access-ptqq2\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:19.541975 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.541972 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a74f616-c592-4caa-9918-b774a56814af-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:19.541975 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.541983 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a74f616-c592-4caa-9918-b774a56814af-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:19.542230 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.541991 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a74f616-c592-4caa-9918-b774a56814af-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:19.546789 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.546765 4947 scope.go:122] "RemoveContainer" containerID="4e1c0285516289527713c98153cfbd396f4e438b85cd4fe228cde0adeeafe9ac" Nov 03 17:31:19.721046 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.720953 4947 generic.go:356] "Generic (PLEG): container finished" podID="6a74f616-c592-4caa-9918-b774a56814af" containerID="f63d19213c2f35c65d339c67ecce2127f681b643d2a8bde4e99a509ed6e8d338" exitCode=0 Nov 03 17:31:19.721046 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.721017 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" Nov 03 17:31:19.721046 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.721036 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" event={"ID":"6a74f616-c592-4caa-9918-b774a56814af","Type":"ContainerDied","Data":"f63d19213c2f35c65d339c67ecce2127f681b643d2a8bde4e99a509ed6e8d338"} Nov 03 17:31:19.721322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.721086 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9" event={"ID":"6a74f616-c592-4caa-9918-b774a56814af","Type":"ContainerDied","Data":"307798dd8f6805ed8415707580e36a405a499c01460a66791e48dff237a00c05"} Nov 03 17:31:19.721322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.721117 4947 scope.go:122] "RemoveContainer" containerID="f63d19213c2f35c65d339c67ecce2127f681b643d2a8bde4e99a509ed6e8d338" Nov 03 17:31:19.733896 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.733875 4947 scope.go:122] "RemoveContainer" containerID="3c1be9ffad043cb343dda78bacb39f45a269709113283270b63745cc6a07c59b" Nov 03 17:31:19.743027 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.743009 4947 scope.go:122] "RemoveContainer" containerID="f63d19213c2f35c65d339c67ecce2127f681b643d2a8bde4e99a509ed6e8d338" Nov 03 17:31:19.743320 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:31:19.743298 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63d19213c2f35c65d339c67ecce2127f681b643d2a8bde4e99a509ed6e8d338\": container with ID starting with f63d19213c2f35c65d339c67ecce2127f681b643d2a8bde4e99a509ed6e8d338 not found: ID does not exist" containerID="f63d19213c2f35c65d339c67ecce2127f681b643d2a8bde4e99a509ed6e8d338" Nov 03 17:31:19.743383 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.743334 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63d19213c2f35c65d339c67ecce2127f681b643d2a8bde4e99a509ed6e8d338"} err="failed to get container status \"f63d19213c2f35c65d339c67ecce2127f681b643d2a8bde4e99a509ed6e8d338\": rpc error: code = NotFound desc = could not find container \"f63d19213c2f35c65d339c67ecce2127f681b643d2a8bde4e99a509ed6e8d338\": container with ID starting with f63d19213c2f35c65d339c67ecce2127f681b643d2a8bde4e99a509ed6e8d338 not found: ID does not exist" Nov 03 17:31:19.743383 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.743357 4947 scope.go:122] "RemoveContainer" containerID="3c1be9ffad043cb343dda78bacb39f45a269709113283270b63745cc6a07c59b" Nov 03 17:31:19.743635 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:31:19.743617 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c1be9ffad043cb343dda78bacb39f45a269709113283270b63745cc6a07c59b\": container with ID starting with 3c1be9ffad043cb343dda78bacb39f45a269709113283270b63745cc6a07c59b not found: ID does not exist" containerID="3c1be9ffad043cb343dda78bacb39f45a269709113283270b63745cc6a07c59b" Nov 03 17:31:19.743685 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.743643 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c1be9ffad043cb343dda78bacb39f45a269709113283270b63745cc6a07c59b"} err="failed to get container status \"3c1be9ffad043cb343dda78bacb39f45a269709113283270b63745cc6a07c59b\": rpc error: code = NotFound desc = could not find container \"3c1be9ffad043cb343dda78bacb39f45a269709113283270b63745cc6a07c59b\": container with ID starting with 3c1be9ffad043cb343dda78bacb39f45a269709113283270b63745cc6a07c59b not found: ID does not exist" Nov 03 17:31:19.778868 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.778845 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9"] Nov 03 17:31:19.787082 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:19.787063 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-06fa9cd7-openshift-default-7759f6cf9c-xjkq9"] Nov 03 17:31:20.290579 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:20.290548 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:31:20.565842 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:20.565767 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a74f616-c592-4caa-9918-b774a56814af" path="/var/lib/kubelet/pods/6a74f616-c592-4caa-9918-b774a56814af/volumes" Nov 03 17:31:24.543530 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.543493 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" containerID="cri-o://a4aeb09df74a8cf0c31f7922c2d50b39f28acb5c91a26f4c0851202d965260f3" gracePeriod=45 Nov 03 17:31:24.755757 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.755718 4947 generic.go:356] "Generic (PLEG): container finished" podID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerID="a4aeb09df74a8cf0c31f7922c2d50b39f28acb5c91a26f4c0851202d965260f3" exitCode=0 Nov 03 17:31:24.755968 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.755796 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" event={"ID":"3a46652a-73e1-4a0e-95ab-668bb2126992","Type":"ContainerDied","Data":"a4aeb09df74a8cf0c31f7922c2d50b39f28acb5c91a26f4c0851202d965260f3"} Nov 03 17:31:24.805654 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.805592 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 17:31:24.909493 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.909458 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a46652a-73e1-4a0e-95ab-668bb2126992-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a46652a-73e1-4a0e-95ab-668bb2126992-tls-certs\") pod \"3a46652a-73e1-4a0e-95ab-668bb2126992\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " Nov 03 17:31:24.909725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.909550 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-dshm\") pod \"3a46652a-73e1-4a0e-95ab-668bb2126992\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " Nov 03 17:31:24.909725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.909608 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-model-cache\") pod \"3a46652a-73e1-4a0e-95ab-668bb2126992\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " Nov 03 17:31:24.909872 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.909786 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a46652a-73e1-4a0e-95ab-668bb2126992-kube-api-access-mlnqv\" (UniqueName: \"kubernetes.io/projected/3a46652a-73e1-4a0e-95ab-668bb2126992-kube-api-access-mlnqv\") pod \"3a46652a-73e1-4a0e-95ab-668bb2126992\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " Nov 03 17:31:24.909945 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.909874 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-home\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-home\") pod \"3a46652a-73e1-4a0e-95ab-668bb2126992\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " Nov 03 17:31:24.910013 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.909934 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-model-cache" pod "3a46652a-73e1-4a0e-95ab-668bb2126992" (UID: "3a46652a-73e1-4a0e-95ab-668bb2126992"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:24.910013 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.909974 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-tmp-dir\") pod \"3a46652a-73e1-4a0e-95ab-668bb2126992\" (UID: \"3a46652a-73e1-4a0e-95ab-668bb2126992\") " Nov 03 17:31:24.910151 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.910131 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-home" pod "3a46652a-73e1-4a0e-95ab-668bb2126992" (UID: "3a46652a-73e1-4a0e-95ab-668bb2126992"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:24.910365 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.910321 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-tmp-dir" pod "3a46652a-73e1-4a0e-95ab-668bb2126992" (UID: "3a46652a-73e1-4a0e-95ab-668bb2126992"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:24.910552 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.910534 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:24.910629 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.910556 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:24.910629 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.910568 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:24.912053 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.911980 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a46652a-73e1-4a0e-95ab-668bb2126992-tls-certs" pod "3a46652a-73e1-4a0e-95ab-668bb2126992" (UID: "3a46652a-73e1-4a0e-95ab-668bb2126992"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:31:24.912524 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.912495 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-dshm" pod "3a46652a-73e1-4a0e-95ab-668bb2126992" (UID: "3a46652a-73e1-4a0e-95ab-668bb2126992"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:24.912818 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:24.912794 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a46652a-73e1-4a0e-95ab-668bb2126992-kube-api-access-mlnqv" pod "3a46652a-73e1-4a0e-95ab-668bb2126992" (UID: "3a46652a-73e1-4a0e-95ab-668bb2126992"). InnerVolumeSpecName "kube-api-access-mlnqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:25.013446 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:25.011897 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlnqv\" (UniqueName: \"kubernetes.io/projected/3a46652a-73e1-4a0e-95ab-668bb2126992-kube-api-access-mlnqv\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.013446 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:25.011932 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a46652a-73e1-4a0e-95ab-668bb2126992-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.013446 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:25.011950 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a46652a-73e1-4a0e-95ab-668bb2126992-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:25.724741 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:25.724684 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:31:25.762612 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:25.762590 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" Nov 03 17:31:25.762783 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:25.762631 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68" event={"ID":"3a46652a-73e1-4a0e-95ab-668bb2126992","Type":"ContainerDied","Data":"6e7f4a8e2946f01d25622df7a97916ad52ae6f363aa0a32fd5513525cddc8771"} Nov 03 17:31:25.762783 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:25.762682 4947 scope.go:122] "RemoveContainer" containerID="a4aeb09df74a8cf0c31f7922c2d50b39f28acb5c91a26f4c0851202d965260f3" Nov 03 17:31:25.817559 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:25.817530 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68"] Nov 03 17:31:25.830564 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:25.830540 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7519b7/distinct-w2-7519b7-kserve-59f85cc7f8-kfl68"] Nov 03 17:31:26.608506 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:26.608465 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" path="/var/lib/kubelet/pods/3a46652a-73e1-4a0e-95ab-668bb2126992/volumes" Nov 03 17:31:30.130630 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:31:30.130550 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:31:42.727109 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:42.727076 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck"] Nov 03 17:31:42.727747 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:42.727353 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" podUID="fd78f22e-864e-4ad2-81bb-4cf0589c3d5f" containerName="maas-api" containerID="cri-o://6baaf7f918673168b39ebf283326ce23d5128d4823d87e82c05dd8619623f8a3" gracePeriod=30 Nov 03 17:31:42.898474 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:42.898433 4947 generic.go:356] "Generic (PLEG): container finished" podID="fd78f22e-864e-4ad2-81bb-4cf0589c3d5f" containerID="6baaf7f918673168b39ebf283326ce23d5128d4823d87e82c05dd8619623f8a3" exitCode=0 Nov 03 17:31:42.898659 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:42.898496 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" event={"ID":"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f","Type":"ContainerDied","Data":"6baaf7f918673168b39ebf283326ce23d5128d4823d87e82c05dd8619623f8a3"} Nov 03 17:31:43.055084 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:43.055057 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" Nov 03 17:31:43.139945 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:43.139908 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-maas-api-tls\") pod \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\" (UID: \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\") " Nov 03 17:31:43.140141 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:43.140044 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-kube-api-access-2hs98\" (UniqueName: \"kubernetes.io/projected/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-kube-api-access-2hs98\") pod \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\" (UID: \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\") " Nov 03 17:31:43.140218 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:43.140156 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-metrics-certs\") pod \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\" (UID: \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\") " Nov 03 17:31:43.140218 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:43.140185 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-openshift-service-ca\") pod \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\" (UID: \"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f\") " Nov 03 17:31:43.140618 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:43.140587 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-openshift-service-ca" pod "fd78f22e-864e-4ad2-81bb-4cf0589c3d5f" (UID: "fd78f22e-864e-4ad2-81bb-4cf0589c3d5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:43.140831 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:43.140751 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:43.143020 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:43.142993 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-maas-api-tls" pod "fd78f22e-864e-4ad2-81bb-4cf0589c3d5f" (UID: "fd78f22e-864e-4ad2-81bb-4cf0589c3d5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:31:43.143140 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:43.143019 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-metrics-certs" pod "fd78f22e-864e-4ad2-81bb-4cf0589c3d5f" (UID: "fd78f22e-864e-4ad2-81bb-4cf0589c3d5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:31:43.143140 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:43.143039 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-kube-api-access-2hs98" pod "fd78f22e-864e-4ad2-81bb-4cf0589c3d5f" (UID: "fd78f22e-864e-4ad2-81bb-4cf0589c3d5f"). InnerVolumeSpecName "kube-api-access-2hs98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:43.255466 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:43.255422 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:43.255777 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:43.255714 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:43.255777 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:43.255755 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hs98\" (UniqueName: \"kubernetes.io/projected/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f-kube-api-access-2hs98\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:43.906267 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:43.906236 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" Nov 03 17:31:43.906786 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:43.906272 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck" event={"ID":"fd78f22e-864e-4ad2-81bb-4cf0589c3d5f","Type":"ContainerDied","Data":"70a90f95fff5467e8093536ed1d8c301e06fc8354ae095a9b663fe2de336cecf"} Nov 03 17:31:43.906786 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:43.906333 4947 scope.go:122] "RemoveContainer" containerID="6baaf7f918673168b39ebf283326ce23d5128d4823d87e82c05dd8619623f8a3" Nov 03 17:31:43.963487 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:43.963460 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck"] Nov 03 17:31:43.975120 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:43.975097 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3dd067-5b6c77ffd5-xp2ck"] Nov 03 17:31:44.565557 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:44.565526 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd78f22e-864e-4ad2-81bb-4cf0589c3d5f" path="/var/lib/kubelet/pods/fd78f22e-864e-4ad2-81bb-4cf0589c3d5f/volumes" Nov 03 17:31:45.658891 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:45.658849 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42"] Nov 03 17:31:45.659398 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:45.659113 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" podUID="2b9c529d-c795-41ac-8145-e126598ecc4e" containerName="istio-proxy" containerID="cri-o://45beecef70d2a31d9418a65d7cfdf1824e004a011a5146761e0fa671afc98d01" gracePeriod=30 Nov 03 17:31:46.391324 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:46.391289 4947 patch_prober.go:29] interesting pod/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:31:46.391531 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:46.391355 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" podUID="2b9c529d-c795-41ac-8145-e126598ecc4e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:31:48.442110 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.442076 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57"] Nov 03 17:31:48.442623 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.442284 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" podUID="a39106eb-e15d-4413-9706-808f2de6d102" containerName="maas-api" containerID="cri-o://646c6d65368375dc78c59e6676512de7a33d690130c40ac10577da2df7c9df92" gracePeriod=30 Nov 03 17:31:48.769252 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.769205 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" Nov 03 17:31:48.844515 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.844482 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a39106eb-e15d-4413-9706-808f2de6d102-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39106eb-e15d-4413-9706-808f2de6d102-metrics-certs\") pod \"a39106eb-e15d-4413-9706-808f2de6d102\" (UID: \"a39106eb-e15d-4413-9706-808f2de6d102\") " Nov 03 17:31:48.844931 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.844910 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39106eb-e15d-4413-9706-808f2de6d102-kube-api-access-wj4wv\" (UniqueName: \"kubernetes.io/projected/a39106eb-e15d-4413-9706-808f2de6d102-kube-api-access-wj4wv\") pod \"a39106eb-e15d-4413-9706-808f2de6d102\" (UID: \"a39106eb-e15d-4413-9706-808f2de6d102\") " Nov 03 17:31:48.845050 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.844956 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a39106eb-e15d-4413-9706-808f2de6d102-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39106eb-e15d-4413-9706-808f2de6d102-openshift-service-ca\") pod \"a39106eb-e15d-4413-9706-808f2de6d102\" (UID: \"a39106eb-e15d-4413-9706-808f2de6d102\") " Nov 03 17:31:48.845050 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.845014 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a39106eb-e15d-4413-9706-808f2de6d102-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39106eb-e15d-4413-9706-808f2de6d102-maas-api-tls\") pod \"a39106eb-e15d-4413-9706-808f2de6d102\" (UID: \"a39106eb-e15d-4413-9706-808f2de6d102\") " Nov 03 17:31:48.846171 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.846043 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39106eb-e15d-4413-9706-808f2de6d102-openshift-service-ca" pod "a39106eb-e15d-4413-9706-808f2de6d102" (UID: "a39106eb-e15d-4413-9706-808f2de6d102"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:48.847380 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.847332 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39106eb-e15d-4413-9706-808f2de6d102-metrics-certs" pod "a39106eb-e15d-4413-9706-808f2de6d102" (UID: "a39106eb-e15d-4413-9706-808f2de6d102"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:31:48.849109 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.849072 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a39106eb-e15d-4413-9706-808f2de6d102-maas-api-tls" pod "a39106eb-e15d-4413-9706-808f2de6d102" (UID: "a39106eb-e15d-4413-9706-808f2de6d102"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:31:48.849710 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.849662 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39106eb-e15d-4413-9706-808f2de6d102-kube-api-access-wj4wv" pod "a39106eb-e15d-4413-9706-808f2de6d102" (UID: "a39106eb-e15d-4413-9706-808f2de6d102"). InnerVolumeSpecName "kube-api-access-wj4wv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:48.972738 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.969192 4947 generic.go:356] "Generic (PLEG): container finished" podID="a39106eb-e15d-4413-9706-808f2de6d102" containerID="646c6d65368375dc78c59e6676512de7a33d690130c40ac10577da2df7c9df92" exitCode=0 Nov 03 17:31:48.972738 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.969376 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" event={"ID":"a39106eb-e15d-4413-9706-808f2de6d102","Type":"ContainerDied","Data":"646c6d65368375dc78c59e6676512de7a33d690130c40ac10577da2df7c9df92"} Nov 03 17:31:48.972738 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.969427 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" event={"ID":"a39106eb-e15d-4413-9706-808f2de6d102","Type":"ContainerDied","Data":"f618a4e834e09f1335727cd380e04db7ffc5f3c8cec17f68bbdc90d5aea08f2f"} Nov 03 17:31:48.972738 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.969465 4947 scope.go:122] "RemoveContainer" containerID="646c6d65368375dc78c59e6676512de7a33d690130c40ac10577da2df7c9df92" Nov 03 17:31:48.972738 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.969656 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57" Nov 03 17:31:48.994791 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.994760 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a39106eb-e15d-4413-9706-808f2de6d102-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:48.996356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.996338 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj4wv\" (UniqueName: \"kubernetes.io/projected/a39106eb-e15d-4413-9706-808f2de6d102-kube-api-access-wj4wv\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:48.996488 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.996473 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a39106eb-e15d-4413-9706-808f2de6d102-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:48.996588 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:48.996577 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a39106eb-e15d-4413-9706-808f2de6d102-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:49.047401 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:49.047375 4947 scope.go:122] "RemoveContainer" containerID="646c6d65368375dc78c59e6676512de7a33d690130c40ac10577da2df7c9df92" Nov 03 17:31:49.047851 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:31:49.047824 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646c6d65368375dc78c59e6676512de7a33d690130c40ac10577da2df7c9df92\": container with ID starting with 646c6d65368375dc78c59e6676512de7a33d690130c40ac10577da2df7c9df92 not found: ID does not exist" containerID="646c6d65368375dc78c59e6676512de7a33d690130c40ac10577da2df7c9df92" Nov 03 17:31:49.047961 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:49.047860 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646c6d65368375dc78c59e6676512de7a33d690130c40ac10577da2df7c9df92"} err="failed to get container status \"646c6d65368375dc78c59e6676512de7a33d690130c40ac10577da2df7c9df92\": rpc error: code = NotFound desc = could not find container \"646c6d65368375dc78c59e6676512de7a33d690130c40ac10577da2df7c9df92\": container with ID starting with 646c6d65368375dc78c59e6676512de7a33d690130c40ac10577da2df7c9df92 not found: ID does not exist" Nov 03 17:31:49.091231 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:49.091187 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57"] Nov 03 17:31:49.096151 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:49.096125 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-7519b7-7f8bbf475b-sdz57"] Nov 03 17:31:50.567408 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:50.567372 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a39106eb-e15d-4413-9706-808f2de6d102" path="/var/lib/kubelet/pods/a39106eb-e15d-4413-9706-808f2de6d102/volumes" Nov 03 17:31:50.957015 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:50.956989 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 17:31:50.995939 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:50.995898 4947 generic.go:356] "Generic (PLEG): container finished" podID="2b9c529d-c795-41ac-8145-e126598ecc4e" containerID="45beecef70d2a31d9418a65d7cfdf1824e004a011a5146761e0fa671afc98d01" exitCode=0 Nov 03 17:31:50.996115 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:50.995945 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" Nov 03 17:31:50.996115 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:50.995983 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" event={"ID":"2b9c529d-c795-41ac-8145-e126598ecc4e","Type":"ContainerDied","Data":"45beecef70d2a31d9418a65d7cfdf1824e004a011a5146761e0fa671afc98d01"} Nov 03 17:31:50.996115 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:50.996050 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42" event={"ID":"2b9c529d-c795-41ac-8145-e126598ecc4e","Type":"ContainerDied","Data":"7791b0c147efb8a0ac153c8400b90fb93e95aaacbb60f1a8cf467e3e61b225b3"} Nov 03 17:31:50.996115 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:50.996079 4947 scope.go:122] "RemoveContainer" containerID="45beecef70d2a31d9418a65d7cfdf1824e004a011a5146761e0fa671afc98d01" Nov 03 17:31:51.006386 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.006369 4947 scope.go:122] "RemoveContainer" containerID="038c2cadb91ff28ef9731c3ea519ce7bf9f90a73dc2da6fa18d29f4e484c933b" Nov 03 17:31:51.015746 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.015727 4947 scope.go:122] "RemoveContainer" containerID="45beecef70d2a31d9418a65d7cfdf1824e004a011a5146761e0fa671afc98d01" Nov 03 17:31:51.016032 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:31:51.016013 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45beecef70d2a31d9418a65d7cfdf1824e004a011a5146761e0fa671afc98d01\": container with ID starting with 45beecef70d2a31d9418a65d7cfdf1824e004a011a5146761e0fa671afc98d01 not found: ID does not exist" containerID="45beecef70d2a31d9418a65d7cfdf1824e004a011a5146761e0fa671afc98d01" Nov 03 17:31:51.016082 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.016043 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45beecef70d2a31d9418a65d7cfdf1824e004a011a5146761e0fa671afc98d01"} err="failed to get container status \"45beecef70d2a31d9418a65d7cfdf1824e004a011a5146761e0fa671afc98d01\": rpc error: code = NotFound desc = could not find container \"45beecef70d2a31d9418a65d7cfdf1824e004a011a5146761e0fa671afc98d01\": container with ID starting with 45beecef70d2a31d9418a65d7cfdf1824e004a011a5146761e0fa671afc98d01 not found: ID does not exist" Nov 03 17:31:51.016082 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.016063 4947 scope.go:122] "RemoveContainer" containerID="038c2cadb91ff28ef9731c3ea519ce7bf9f90a73dc2da6fa18d29f4e484c933b" Nov 03 17:31:51.016286 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:31:51.016271 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"038c2cadb91ff28ef9731c3ea519ce7bf9f90a73dc2da6fa18d29f4e484c933b\": container with ID starting with 038c2cadb91ff28ef9731c3ea519ce7bf9f90a73dc2da6fa18d29f4e484c933b not found: ID does not exist" containerID="038c2cadb91ff28ef9731c3ea519ce7bf9f90a73dc2da6fa18d29f4e484c933b" Nov 03 17:31:51.016338 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.016289 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038c2cadb91ff28ef9731c3ea519ce7bf9f90a73dc2da6fa18d29f4e484c933b"} err="failed to get container status \"038c2cadb91ff28ef9731c3ea519ce7bf9f90a73dc2da6fa18d29f4e484c933b\": rpc error: code = NotFound desc = could not find container \"038c2cadb91ff28ef9731c3ea519ce7bf9f90a73dc2da6fa18d29f4e484c933b\": container with ID starting with 038c2cadb91ff28ef9731c3ea519ce7bf9f90a73dc2da6fa18d29f4e484c933b not found: ID does not exist" Nov 03 17:31:51.018858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.018800 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-credential-socket\") pod \"2b9c529d-c795-41ac-8145-e126598ecc4e\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " Nov 03 17:31:51.019058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.018857 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-envoy\") pod \"2b9c529d-c795-41ac-8145-e126598ecc4e\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " Nov 03 17:31:51.019058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.018884 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-data\") pod \"2b9c529d-c795-41ac-8145-e126598ecc4e\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " Nov 03 17:31:51.019058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.018934 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-token\" (UniqueName: \"kubernetes.io/projected/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-token\") pod \"2b9c529d-c795-41ac-8145-e126598ecc4e\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " Nov 03 17:31:51.019058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.019029 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-workload-socket\") pod \"2b9c529d-c795-41ac-8145-e126598ecc4e\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " Nov 03 17:31:51.019287 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.019059 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-workload-certs\") pod \"2b9c529d-c795-41ac-8145-e126598ecc4e\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " Nov 03 17:31:51.019287 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.019060 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-credential-socket" pod "2b9c529d-c795-41ac-8145-e126598ecc4e" (UID: "2b9c529d-c795-41ac-8145-e126598ecc4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:51.019287 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.019137 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b9c529d-c795-41ac-8145-e126598ecc4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9c529d-c795-41ac-8145-e126598ecc4e-istiod-ca-cert\") pod \"2b9c529d-c795-41ac-8145-e126598ecc4e\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " Nov 03 17:31:51.019287 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.019181 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9c529d-c795-41ac-8145-e126598ecc4e-kube-api-access-5cwx9\" (UniqueName: \"kubernetes.io/projected/2b9c529d-c795-41ac-8145-e126598ecc4e-kube-api-access-5cwx9\") pod \"2b9c529d-c795-41ac-8145-e126598ecc4e\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " Nov 03 17:31:51.019287 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.019196 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-data" pod "2b9c529d-c795-41ac-8145-e126598ecc4e" (UID: "2b9c529d-c795-41ac-8145-e126598ecc4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:51.019287 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.019259 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-podinfo\") pod \"2b9c529d-c795-41ac-8145-e126598ecc4e\" (UID: \"2b9c529d-c795-41ac-8145-e126598ecc4e\") " Nov 03 17:31:51.019929 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.019879 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:51.019929 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.019904 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:51.020421 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.020268 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9c529d-c795-41ac-8145-e126598ecc4e-istiod-ca-cert" pod "2b9c529d-c795-41ac-8145-e126598ecc4e" (UID: "2b9c529d-c795-41ac-8145-e126598ecc4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:31:51.021309 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.021262 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-workload-socket" pod "2b9c529d-c795-41ac-8145-e126598ecc4e" (UID: "2b9c529d-c795-41ac-8145-e126598ecc4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:51.021416 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.019463 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-workload-certs" pod "2b9c529d-c795-41ac-8145-e126598ecc4e" (UID: "2b9c529d-c795-41ac-8145-e126598ecc4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:51.022792 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.022689 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-token" pod "2b9c529d-c795-41ac-8145-e126598ecc4e" (UID: "2b9c529d-c795-41ac-8145-e126598ecc4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:51.023208 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.023183 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-podinfo" pod "2b9c529d-c795-41ac-8145-e126598ecc4e" (UID: "2b9c529d-c795-41ac-8145-e126598ecc4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:31:51.024124 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.024097 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-envoy" pod "2b9c529d-c795-41ac-8145-e126598ecc4e" (UID: "2b9c529d-c795-41ac-8145-e126598ecc4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:31:51.024247 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.024187 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9c529d-c795-41ac-8145-e126598ecc4e-kube-api-access-5cwx9" pod "2b9c529d-c795-41ac-8145-e126598ecc4e" (UID: "2b9c529d-c795-41ac-8145-e126598ecc4e"). InnerVolumeSpecName "kube-api-access-5cwx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:31:51.120973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.120885 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:51.120973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.120920 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b9c529d-c795-41ac-8145-e126598ecc4e-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:51.120973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.120933 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cwx9\" (UniqueName: \"kubernetes.io/projected/2b9c529d-c795-41ac-8145-e126598ecc4e-kube-api-access-5cwx9\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:51.120973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.120951 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:51.120973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.120964 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:51.120973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.120976 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b9c529d-c795-41ac-8145-e126598ecc4e-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:51.121332 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.120987 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b9c529d-c795-41ac-8145-e126598ecc4e-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:31:51.345561 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.345525 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42"] Nov 03 17:31:51.358669 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:51.358641 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-3dd067-openshift-default-5f9777b485-mmb42"] Nov 03 17:31:52.566904 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:52.566873 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b9c529d-c795-41ac-8145-e126598ecc4e" path="/var/lib/kubelet/pods/2b9c529d-c795-41ac-8145-e126598ecc4e/volumes" Nov 03 17:31:53.414473 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.413033 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8"] Nov 03 17:31:53.415306 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.415280 4947 cpu_manager.go:414] "containerMap: removing container" podUID="9644e7c0-4d23-498e-8a40-e76349d2e468" containerName="cleanup" Nov 03 17:31:53.415491 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.415474 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9644e7c0-4d23-498e-8a40-e76349d2e468" containerName="cleanup" Nov 03 17:31:53.415617 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.415603 4947 cpu_manager.go:414] "containerMap: removing container" podUID="cf0d4db9-e143-40e6-a677-2c2ea454ac3b" containerName="payload-pre-processing" Nov 03 17:31:53.415745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.415729 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf0d4db9-e143-40e6-a677-2c2ea454ac3b" containerName="payload-pre-processing" Nov 03 17:31:53.415858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.415845 4947 cpu_manager.go:414] "containerMap: removing container" podUID="bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7" containerName="cleanup" Nov 03 17:31:53.415986 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.415971 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7" containerName="cleanup" Nov 03 17:31:53.416095 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.416083 4947 cpu_manager.go:414] "containerMap: removing container" podUID="cab0d416-6e99-420f-9730-2e3b654e354f" containerName="maas-api" Nov 03 17:31:53.416187 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.416175 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab0d416-6e99-420f-9730-2e3b654e354f" containerName="maas-api" Nov 03 17:31:53.416284 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.416273 4947 cpu_manager.go:414] "containerMap: removing container" podUID="e214eab6-0f39-415b-8ea9-8f583801475a" containerName="payload-processing" Nov 03 17:31:53.416371 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.416360 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e214eab6-0f39-415b-8ea9-8f583801475a" containerName="payload-processing" Nov 03 17:31:53.416470 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.416459 4947 cpu_manager.go:414] "containerMap: removing container" podUID="5af508d5-80f2-4f5a-b960-bbc2d604e97f" containerName="cleanup" Nov 03 17:31:53.416578 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.416565 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af508d5-80f2-4f5a-b960-bbc2d604e97f" containerName="cleanup" Nov 03 17:31:53.416669 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.416657 4947 cpu_manager.go:414] "containerMap: removing container" podUID="d68c016e-91bf-4967-9d14-c6a0bea44dc4" containerName="istio-proxy" Nov 03 17:31:53.416795 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.416780 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d68c016e-91bf-4967-9d14-c6a0bea44dc4" containerName="istio-proxy" Nov 03 17:31:53.416887 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.416877 4947 cpu_manager.go:414] "containerMap: removing container" podUID="0c92bab9-7a09-4cc8-b4a9-51bd85bfd818" containerName="cleanup" Nov 03 17:31:53.416972 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.416960 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c92bab9-7a09-4cc8-b4a9-51bd85bfd818" containerName="cleanup" Nov 03 17:31:53.417068 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.417056 4947 cpu_manager.go:414] "containerMap: removing container" podUID="aa47da19-7ea1-4601-bfba-18eca9086932" containerName="cleanup" Nov 03 17:31:53.417147 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.417136 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa47da19-7ea1-4601-bfba-18eca9086932" containerName="cleanup" Nov 03 17:31:53.417240 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.417229 4947 cpu_manager.go:414] "containerMap: removing container" podUID="01cb2467-7a13-4f96-9c67-2cc5104d99b9" containerName="cleanup" Nov 03 17:31:53.417325 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.417314 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01cb2467-7a13-4f96-9c67-2cc5104d99b9" containerName="cleanup" Nov 03 17:31:53.417408 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.417398 4947 cpu_manager.go:414] "containerMap: removing container" podUID="0c92bab9-7a09-4cc8-b4a9-51bd85bfd818" containerName="cleanup" Nov 03 17:31:53.417494 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.417483 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c92bab9-7a09-4cc8-b4a9-51bd85bfd818" containerName="cleanup" Nov 03 17:31:53.417581 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.417571 4947 cpu_manager.go:414] "containerMap: removing container" podUID="01cb2467-7a13-4f96-9c67-2cc5104d99b9" containerName="cleanup" Nov 03 17:31:53.417661 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.417650 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01cb2467-7a13-4f96-9c67-2cc5104d99b9" containerName="cleanup" Nov 03 17:31:53.417773 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.417761 4947 cpu_manager.go:414] "containerMap: removing container" podUID="53960065-d46f-4926-82c0-6b180e9eab56" containerName="cleanup" Nov 03 17:31:53.417864 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.417852 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53960065-d46f-4926-82c0-6b180e9eab56" containerName="cleanup" Nov 03 17:31:53.417950 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.417940 4947 cpu_manager.go:414] "containerMap: removing container" podUID="9644e7c0-4d23-498e-8a40-e76349d2e468" containerName="cleanup" Nov 03 17:31:53.418044 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418024 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9644e7c0-4d23-498e-8a40-e76349d2e468" containerName="cleanup" Nov 03 17:31:53.418044 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418048 4947 cpu_manager.go:414] "containerMap: removing container" podUID="f6e02969-c94d-47bf-9b80-278e570b6e0c" containerName="istio-proxy" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418056 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e02969-c94d-47bf-9b80-278e570b6e0c" containerName="istio-proxy" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418071 4947 cpu_manager.go:414] "containerMap: removing container" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418078 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418089 4947 cpu_manager.go:414] "containerMap: removing container" podUID="bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7" containerName="cleanup" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418097 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7" containerName="cleanup" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418108 4947 cpu_manager.go:414] "containerMap: removing container" podUID="53960065-d46f-4926-82c0-6b180e9eab56" containerName="cleanup" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418114 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53960065-d46f-4926-82c0-6b180e9eab56" containerName="cleanup" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418126 4947 cpu_manager.go:414] "containerMap: removing container" podUID="2b9c529d-c795-41ac-8145-e126598ecc4e" containerName="istio-proxy" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418134 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9c529d-c795-41ac-8145-e126598ecc4e" containerName="istio-proxy" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418144 4947 cpu_manager.go:414] "containerMap: removing container" podUID="5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" containerName="istio-proxy" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418151 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" containerName="istio-proxy" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418161 4947 cpu_manager.go:414] "containerMap: removing container" podUID="fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8" containerName="maas-api" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418167 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8" containerName="maas-api" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418182 4947 cpu_manager.go:414] "containerMap: removing container" podUID="f6e02969-c94d-47bf-9b80-278e570b6e0c" containerName="istio-proxy" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418190 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e02969-c94d-47bf-9b80-278e570b6e0c" containerName="istio-proxy" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418199 4947 cpu_manager.go:414] "containerMap: removing container" podUID="aa47da19-7ea1-4601-bfba-18eca9086932" containerName="cleanup" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418206 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa47da19-7ea1-4601-bfba-18eca9086932" containerName="cleanup" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418218 4947 cpu_manager.go:414] "containerMap: removing container" podUID="bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7" containerName="cleanup" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418225 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7" containerName="cleanup" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418237 4947 cpu_manager.go:414] "containerMap: removing container" podUID="fd78f22e-864e-4ad2-81bb-4cf0589c3d5f" containerName="maas-api" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418245 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd78f22e-864e-4ad2-81bb-4cf0589c3d5f" containerName="maas-api" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418255 4947 cpu_manager.go:414] "containerMap: removing container" podUID="9644e7c0-4d23-498e-8a40-e76349d2e468" containerName="cleanup" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418262 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9644e7c0-4d23-498e-8a40-e76349d2e468" containerName="cleanup" Nov 03 17:31:53.418261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418276 4947 cpu_manager.go:414] "containerMap: removing container" podUID="6a74f616-c592-4caa-9918-b774a56814af" containerName="istio-proxy" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418283 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a74f616-c592-4caa-9918-b774a56814af" containerName="istio-proxy" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418305 4947 cpu_manager.go:414] "containerMap: removing container" podUID="a39106eb-e15d-4413-9706-808f2de6d102" containerName="maas-api" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418312 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39106eb-e15d-4413-9706-808f2de6d102" containerName="maas-api" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418325 4947 cpu_manager.go:414] "containerMap: removing container" podUID="6a74f616-c592-4caa-9918-b774a56814af" containerName="istio-proxy" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418332 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a74f616-c592-4caa-9918-b774a56814af" containerName="istio-proxy" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418343 4947 cpu_manager.go:414] "containerMap: removing container" podUID="2b9c529d-c795-41ac-8145-e126598ecc4e" containerName="istio-proxy" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418350 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9c529d-c795-41ac-8145-e126598ecc4e" containerName="istio-proxy" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418367 4947 cpu_manager.go:414] "containerMap: removing container" podUID="0c92bab9-7a09-4cc8-b4a9-51bd85bfd818" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418374 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c92bab9-7a09-4cc8-b4a9-51bd85bfd818" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418389 4947 cpu_manager.go:414] "containerMap: removing container" podUID="97f5d9a3-9c9e-404f-a572-7821d58bcb8d" containerName="payload-processing" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418397 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f5d9a3-9c9e-404f-a572-7821d58bcb8d" containerName="payload-processing" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418413 4947 cpu_manager.go:414] "containerMap: removing container" podUID="5af508d5-80f2-4f5a-b960-bbc2d604e97f" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418419 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af508d5-80f2-4f5a-b960-bbc2d604e97f" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418434 4947 cpu_manager.go:414] "containerMap: removing container" podUID="5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" containerName="istio-proxy" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418441 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" containerName="istio-proxy" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418456 4947 cpu_manager.go:414] "containerMap: removing container" podUID="53960065-d46f-4926-82c0-6b180e9eab56" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418463 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53960065-d46f-4926-82c0-6b180e9eab56" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418638 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39106eb-e15d-4413-9706-808f2de6d102" containerName="maas-api" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418655 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a74f616-c592-4caa-9918-b774a56814af" containerName="istio-proxy" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418665 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9c529d-c795-41ac-8145-e126598ecc4e" containerName="istio-proxy" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418675 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa47da19-7ea1-4601-bfba-18eca9086932" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418684 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c92bab9-7a09-4cc8-b4a9-51bd85bfd818" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418713 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f5d9a3-9c9e-404f-a572-7821d58bcb8d" containerName="payload-processing" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418725 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af508d5-80f2-4f5a-b960-bbc2d604e97f" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418735 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" containerName="istio-proxy" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418745 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="53960065-d46f-4926-82c0-6b180e9eab56" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418753 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9644e7c0-4d23-498e-8a40-e76349d2e468" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418763 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf0d4db9-e143-40e6-a677-2c2ea454ac3b" containerName="payload-pre-processing" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418775 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418784 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab0d416-6e99-420f-9730-2e3b654e354f" containerName="maas-api" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418794 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="d68c016e-91bf-4967-9d14-c6a0bea44dc4" containerName="istio-proxy" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418803 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af508d5-80f2-4f5a-b960-bbc2d604e97f" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418811 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e214eab6-0f39-415b-8ea9-8f583801475a" containerName="payload-processing" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418820 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c92bab9-7a09-4cc8-b4a9-51bd85bfd818" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418828 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="01cb2467-7a13-4f96-9c67-2cc5104d99b9" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418836 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="01cb2467-7a13-4f96-9c67-2cc5104d99b9" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418845 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="53960065-d46f-4926-82c0-6b180e9eab56" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418856 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e02969-c94d-47bf-9b80-278e570b6e0c" containerName="istio-proxy" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418864 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a46652a-73e1-4a0e-95ab-668bb2126992" containerName="main" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418874 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418882 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="53960065-d46f-4926-82c0-6b180e9eab56" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418891 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9c529d-c795-41ac-8145-e126598ecc4e" containerName="istio-proxy" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418898 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcb7fe1d-f91b-4ef4-8f5b-aa8577c264c8" containerName="maas-api" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418908 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e02969-c94d-47bf-9b80-278e570b6e0c" containerName="istio-proxy" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418916 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa47da19-7ea1-4601-bfba-18eca9086932" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418923 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="01cb2467-7a13-4f96-9c67-2cc5104d99b9" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418932 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbe6d1c1-8bd2-44b4-af60-02a3fe977aa7" containerName="cleanup" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418941 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd78f22e-864e-4ad2-81bb-4cf0589c3d5f" containerName="maas-api" Nov 03 17:31:53.419748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.418951 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9644e7c0-4d23-498e-8a40-e76349d2e468" containerName="cleanup" Nov 03 17:31:53.426313 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.426291 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.452570 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.451527 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8"] Nov 03 17:31:53.453497 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.453469 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-1e9d48\"/\"rate-test-model-1e9d48-kserve-self-signed-certs\"" Nov 03 17:31:53.453608 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.453574 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-1e9d48\"/\"openshift-service-ca.crt\"" Nov 03 17:31:53.454380 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.454355 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-model-cache\") pod \"rate-test-model-1e9d48-kserve-979559c77-z5mf8\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.454497 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.454400 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-tls-certs\") pod \"rate-test-model-1e9d48-kserve-979559c77-z5mf8\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.454497 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.454434 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-dshm\") pod \"rate-test-model-1e9d48-kserve-979559c77-z5mf8\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.454497 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.454467 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqmtd\" (UniqueName: \"kubernetes.io/projected/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-kube-api-access-dqmtd\") pod \"rate-test-model-1e9d48-kserve-979559c77-z5mf8\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.454677 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.454499 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-home\") pod \"rate-test-model-1e9d48-kserve-979559c77-z5mf8\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.454677 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.454541 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-tmp-dir\") pod \"rate-test-model-1e9d48-kserve-979559c77-z5mf8\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.460073 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.460052 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-1e9d48\"/\"default-dockercfg-g9w7v\"" Nov 03 17:31:53.460189 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.460133 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-1e9d48\"/\"kube-root-ca.crt\"" Nov 03 17:31:53.556469 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.556429 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-model-cache\") pod \"rate-test-model-1e9d48-kserve-979559c77-z5mf8\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.556685 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.556484 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-tls-certs\") pod \"rate-test-model-1e9d48-kserve-979559c77-z5mf8\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.556685 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.556519 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-dshm\") pod \"rate-test-model-1e9d48-kserve-979559c77-z5mf8\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.556685 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.556550 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqmtd\" (UniqueName: \"kubernetes.io/projected/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-kube-api-access-dqmtd\") pod \"rate-test-model-1e9d48-kserve-979559c77-z5mf8\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.556685 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.556583 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-home\") pod \"rate-test-model-1e9d48-kserve-979559c77-z5mf8\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.556685 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.556627 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-tmp-dir\") pod \"rate-test-model-1e9d48-kserve-979559c77-z5mf8\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.557029 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.556925 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-model-cache\") pod \"rate-test-model-1e9d48-kserve-979559c77-z5mf8\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.557115 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.557093 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-tmp-dir\") pod \"rate-test-model-1e9d48-kserve-979559c77-z5mf8\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.557231 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.557212 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-home\") pod \"rate-test-model-1e9d48-kserve-979559c77-z5mf8\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.559640 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.559613 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-tls-certs\") pod \"rate-test-model-1e9d48-kserve-979559c77-z5mf8\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.560339 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.560313 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-dshm\") pod \"rate-test-model-1e9d48-kserve-979559c77-z5mf8\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.581105 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.581079 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqmtd\" (UniqueName: \"kubernetes.io/projected/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-kube-api-access-dqmtd\") pod \"rate-test-model-1e9d48-kserve-979559c77-z5mf8\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:53.758725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:53.758673 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:54.069172 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:54.069141 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8"] Nov 03 17:31:54.069667 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:31:54.069642 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb48ab6d4_f151_459b_84a6_fede7e0e6d9c.slice/crio-7062b7d6661f4a405f8c8f9ae08f8fb99577479fce3f75f10257a5c34bb164d5 WatchSource:0}: Error finding container 7062b7d6661f4a405f8c8f9ae08f8fb99577479fce3f75f10257a5c34bb164d5: Status 404 returned error can't find the container with id 7062b7d6661f4a405f8c8f9ae08f8fb99577479fce3f75f10257a5c34bb164d5 Nov 03 17:31:55.023363 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:55.023331 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" event={"ID":"b48ab6d4-f151-459b-84a6-fede7e0e6d9c","Type":"ContainerStarted","Data":"0cc0bfa0ddfb837f992d758c3d97b53626e48a6b9f97e48a4e9c7e0090c50989"} Nov 03 17:31:55.023895 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:55.023373 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" event={"ID":"b48ab6d4-f151-459b-84a6-fede7e0e6d9c","Type":"ContainerStarted","Data":"7062b7d6661f4a405f8c8f9ae08f8fb99577479fce3f75f10257a5c34bb164d5"} Nov 03 17:31:55.078530 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:55.074613 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" podStartSLOduration=2.074590346 podStartE2EDuration="2.074590346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:31:55.069867937 +0000 UTC m=+4658.658867906" watchObservedRunningTime="2026-11-03 17:31:55.074590346 +0000 UTC m=+4658.663590321" Nov 03 17:31:55.759075 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:55.759035 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:31:57.079949 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:57.079908 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk"] Nov 03 17:31:57.080468 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:31:57.080118 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" podUID="ee442df4-03ea-4f56-9b1a-a31e84d2c861" containerName="istio-proxy" containerID="cri-o://02d52b1bdea1129c16dbce0bf1bec3054d85f67bf104fa9e1389a6232ef272d4" gracePeriod=30 Nov 03 17:32:00.129572 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:32:00.129535 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:32:02.347046 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.346979 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 17:32:02.370377 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.370343 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-podinfo\") pod \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " Nov 03 17:32:02.370574 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.370396 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-credential-socket\") pod \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " Nov 03 17:32:02.370574 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.370427 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-data\") pod \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " Nov 03 17:32:02.370574 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.370490 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-workload-socket\") pod \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " Nov 03 17:32:02.370574 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.370524 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-envoy\") pod \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " Nov 03 17:32:02.370881 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.370577 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-workload-certs\") pod \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " Nov 03 17:32:02.370881 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.370637 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istiod-ca-cert\") pod \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " Nov 03 17:32:02.370881 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.370757 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee442df4-03ea-4f56-9b1a-a31e84d2c861-kube-api-access-pttrw\" (UniqueName: \"kubernetes.io/projected/ee442df4-03ea-4f56-9b1a-a31e84d2c861-kube-api-access-pttrw\") pod \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " Nov 03 17:32:02.370881 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.370756 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-credential-socket" pod "ee442df4-03ea-4f56-9b1a-a31e84d2c861" (UID: "ee442df4-03ea-4f56-9b1a-a31e84d2c861"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:02.370881 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.370783 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-data" pod "ee442df4-03ea-4f56-9b1a-a31e84d2c861" (UID: "ee442df4-03ea-4f56-9b1a-a31e84d2c861"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:02.370881 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.370803 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-token\" (UniqueName: \"kubernetes.io/projected/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-token\") pod \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\" (UID: \"ee442df4-03ea-4f56-9b1a-a31e84d2c861\") " Nov 03 17:32:02.371220 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.370947 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-workload-socket" pod "ee442df4-03ea-4f56-9b1a-a31e84d2c861" (UID: "ee442df4-03ea-4f56-9b1a-a31e84d2c861"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:02.371220 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.370966 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-workload-certs" pod "ee442df4-03ea-4f56-9b1a-a31e84d2c861" (UID: "ee442df4-03ea-4f56-9b1a-a31e84d2c861"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:02.371220 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.371096 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istiod-ca-cert" pod "ee442df4-03ea-4f56-9b1a-a31e84d2c861" (UID: "ee442df4-03ea-4f56-9b1a-a31e84d2c861"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:32:02.371643 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.371485 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:02.371643 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.371645 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:02.371901 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.371660 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:02.371901 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.371673 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:02.371901 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.371689 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:02.373090 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.373064 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-podinfo" pod "ee442df4-03ea-4f56-9b1a-a31e84d2c861" (UID: "ee442df4-03ea-4f56-9b1a-a31e84d2c861"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:32:02.373202 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.373183 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee442df4-03ea-4f56-9b1a-a31e84d2c861-kube-api-access-pttrw" pod "ee442df4-03ea-4f56-9b1a-a31e84d2c861" (UID: "ee442df4-03ea-4f56-9b1a-a31e84d2c861"). InnerVolumeSpecName "kube-api-access-pttrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:02.373253 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.373198 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-envoy" pod "ee442df4-03ea-4f56-9b1a-a31e84d2c861" (UID: "ee442df4-03ea-4f56-9b1a-a31e84d2c861"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:02.373253 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.373226 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-token" pod "ee442df4-03ea-4f56-9b1a-a31e84d2c861" (UID: "ee442df4-03ea-4f56-9b1a-a31e84d2c861"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:02.473177 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.473147 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:02.473177 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.473175 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pttrw\" (UniqueName: \"kubernetes.io/projected/ee442df4-03ea-4f56-9b1a-a31e84d2c861-kube-api-access-pttrw\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:02.473357 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.473188 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:02.473357 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:02.473196 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee442df4-03ea-4f56-9b1a-a31e84d2c861-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:03.076213 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:03.076160 4947 generic.go:356] "Generic (PLEG): container finished" podID="ee442df4-03ea-4f56-9b1a-a31e84d2c861" containerID="02d52b1bdea1129c16dbce0bf1bec3054d85f67bf104fa9e1389a6232ef272d4" exitCode=0 Nov 03 17:32:03.076416 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:03.076230 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" event={"ID":"ee442df4-03ea-4f56-9b1a-a31e84d2c861","Type":"ContainerDied","Data":"02d52b1bdea1129c16dbce0bf1bec3054d85f67bf104fa9e1389a6232ef272d4"} Nov 03 17:32:03.076416 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:03.076280 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" event={"ID":"ee442df4-03ea-4f56-9b1a-a31e84d2c861","Type":"ContainerDied","Data":"e192172f142ca83b6efbd489ef6c1ab591ee373f6e40eb5a3b30dd2c6b0a7caa"} Nov 03 17:32:03.076416 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:03.076301 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk" Nov 03 17:32:03.076416 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:03.076306 4947 scope.go:122] "RemoveContainer" containerID="02d52b1bdea1129c16dbce0bf1bec3054d85f67bf104fa9e1389a6232ef272d4" Nov 03 17:32:03.087643 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:03.087622 4947 scope.go:122] "RemoveContainer" containerID="02d52b1bdea1129c16dbce0bf1bec3054d85f67bf104fa9e1389a6232ef272d4" Nov 03 17:32:03.088004 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:32:03.087968 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d52b1bdea1129c16dbce0bf1bec3054d85f67bf104fa9e1389a6232ef272d4\": container with ID starting with 02d52b1bdea1129c16dbce0bf1bec3054d85f67bf104fa9e1389a6232ef272d4 not found: ID does not exist" containerID="02d52b1bdea1129c16dbce0bf1bec3054d85f67bf104fa9e1389a6232ef272d4" Nov 03 17:32:03.088114 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:03.088017 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d52b1bdea1129c16dbce0bf1bec3054d85f67bf104fa9e1389a6232ef272d4"} err="failed to get container status \"02d52b1bdea1129c16dbce0bf1bec3054d85f67bf104fa9e1389a6232ef272d4\": rpc error: code = NotFound desc = could not find container \"02d52b1bdea1129c16dbce0bf1bec3054d85f67bf104fa9e1389a6232ef272d4\": container with ID starting with 02d52b1bdea1129c16dbce0bf1bec3054d85f67bf104fa9e1389a6232ef272d4 not found: ID does not exist" Nov 03 17:32:03.147406 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:03.147369 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk"] Nov 03 17:32:03.160608 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:03.160578 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7519b7-openshift-default-58758fb8cf-lt8tk"] Nov 03 17:32:03.759764 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:03.759725 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:32:03.772042 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:03.772016 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:32:04.098534 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:04.098451 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:32:04.565728 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:04.565653 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee442df4-03ea-4f56-9b1a-a31e84d2c861" path="/var/lib/kubelet/pods/ee442df4-03ea-4f56-9b1a-a31e84d2c861/volumes" Nov 03 17:32:06.933853 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:06.933819 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d"] Nov 03 17:32:13.492399 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:13.492366 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:32:21.955498 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:21.955450 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" containerID="cri-o://01b816830c8070fa5291bed22a0475999fee19402ea5c8258bcf1684959d4b41" gracePeriod=45 Nov 03 17:32:22.224992 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.224906 4947 generic.go:356] "Generic (PLEG): container finished" podID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerID="01b816830c8070fa5291bed22a0475999fee19402ea5c8258bcf1684959d4b41" exitCode=0 Nov 03 17:32:22.225318 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.224992 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" event={"ID":"aa6e9f4a-d932-4a88-b640-6676db6c29f8","Type":"ContainerDied","Data":"01b816830c8070fa5291bed22a0475999fee19402ea5c8258bcf1684959d4b41"} Nov 03 17:32:22.257105 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.257083 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 17:32:22.421424 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.421381 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-dshm\") pod \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " Nov 03 17:32:22.421619 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.421481 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa6e9f4a-d932-4a88-b640-6676db6c29f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6e9f4a-d932-4a88-b640-6676db6c29f8-tls-certs\") pod \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " Nov 03 17:32:22.421619 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.421523 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-home\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-home\") pod \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " Nov 03 17:32:22.421619 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.421566 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-tmp-dir\") pod \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " Nov 03 17:32:22.421856 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.421643 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-model-cache\") pod \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " Nov 03 17:32:22.421856 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.421816 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-tmp-dir" pod "aa6e9f4a-d932-4a88-b640-6676db6c29f8" (UID: "aa6e9f4a-d932-4a88-b640-6676db6c29f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:22.421856 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.421828 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-home" pod "aa6e9f4a-d932-4a88-b640-6676db6c29f8" (UID: "aa6e9f4a-d932-4a88-b640-6676db6c29f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:22.421856 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.421830 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa6e9f4a-d932-4a88-b640-6676db6c29f8-kube-api-access-25kwb\" (UniqueName: \"kubernetes.io/projected/aa6e9f4a-d932-4a88-b640-6676db6c29f8-kube-api-access-25kwb\") pod \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\" (UID: \"aa6e9f4a-d932-4a88-b640-6676db6c29f8\") " Nov 03 17:32:22.422076 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.421926 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-model-cache" pod "aa6e9f4a-d932-4a88-b640-6676db6c29f8" (UID: "aa6e9f4a-d932-4a88-b640-6676db6c29f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:22.422363 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.422343 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.422435 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.422364 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.422435 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.422372 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.423832 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.423806 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-dshm" pod "aa6e9f4a-d932-4a88-b640-6676db6c29f8" (UID: "aa6e9f4a-d932-4a88-b640-6676db6c29f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:32:22.423952 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.423848 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6e9f4a-d932-4a88-b640-6676db6c29f8-tls-certs" pod "aa6e9f4a-d932-4a88-b640-6676db6c29f8" (UID: "aa6e9f4a-d932-4a88-b640-6676db6c29f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:32:22.423952 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.423922 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6e9f4a-d932-4a88-b640-6676db6c29f8-kube-api-access-25kwb" pod "aa6e9f4a-d932-4a88-b640-6676db6c29f8" (UID: "aa6e9f4a-d932-4a88-b640-6676db6c29f8"). InnerVolumeSpecName "kube-api-access-25kwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:22.523149 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.523108 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa6e9f4a-d932-4a88-b640-6676db6c29f8-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.523149 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.523138 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa6e9f4a-d932-4a88-b640-6676db6c29f8-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:22.523149 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:22.523154 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25kwb\" (UniqueName: \"kubernetes.io/projected/aa6e9f4a-d932-4a88-b640-6676db6c29f8-kube-api-access-25kwb\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:23.232469 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:23.232438 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" Nov 03 17:32:23.233067 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:23.232461 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d" event={"ID":"aa6e9f4a-d932-4a88-b640-6676db6c29f8","Type":"ContainerDied","Data":"b60982178bce997a5bc28db77a3203b8e792c8a86ca0b04a4bb9a9d5b526f30f"} Nov 03 17:32:23.233067 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:23.232515 4947 scope.go:122] "RemoveContainer" containerID="01b816830c8070fa5291bed22a0475999fee19402ea5c8258bcf1684959d4b41" Nov 03 17:32:23.291134 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:23.291103 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d"] Nov 03 17:32:23.293981 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:23.293959 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-3a2fa2/premium-w3-3a2fa2-kserve-76bf569f78-pcq5d"] Nov 03 17:32:24.566614 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:24.566583 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" path="/var/lib/kubelet/pods/aa6e9f4a-d932-4a88-b640-6676db6c29f8/volumes" Nov 03 17:32:30.129917 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:32:30.129865 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:32:32.385726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:32.385677 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8"] Nov 03 17:32:32.386196 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:32.385922 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" podUID="7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b" containerName="maas-api" containerID="cri-o://49fb4eeeb4e85aa99e99c9a443c9aacffba382baba0a1eef04feba05fee186fe" gracePeriod=30 Nov 03 17:32:32.700272 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:32.700245 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 17:32:32.771251 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:32.771138 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-metrics-certs\") pod \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\" (UID: \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\") " Nov 03 17:32:32.772381 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:32.772334 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-maas-api-tls\") pod \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\" (UID: \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\") " Nov 03 17:32:32.772381 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:32.772507 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-openshift-service-ca\") pod \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\" (UID: \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\") " Nov 03 17:32:32.773862 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:32.772669 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-kube-api-access-8tzml\" (UniqueName: \"kubernetes.io/projected/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-kube-api-access-8tzml\") pod \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\" (UID: \"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b\") " Nov 03 17:32:32.773968 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:32.773944 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-openshift-service-ca" pod "7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b" (UID: "7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:32:32.815864 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:32.814677 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-maas-api-tls" pod "7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b" (UID: "7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:32:32.826508 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:32.820920 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-kube-api-access-8tzml" pod "7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b" (UID: "7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b"). InnerVolumeSpecName "kube-api-access-8tzml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:32:32.826508 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:32.825851 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-metrics-certs" pod "7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b" (UID: "7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:32:32.874980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:32.874945 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tzml\" (UniqueName: \"kubernetes.io/projected/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-kube-api-access-8tzml\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:32.874980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:32.874976 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:32.874980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:32.874987 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:32.875236 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:32.874996 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:32:33.293779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:33.293746 4947 generic.go:356] "Generic (PLEG): container finished" podID="7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b" containerID="49fb4eeeb4e85aa99e99c9a443c9aacffba382baba0a1eef04feba05fee186fe" exitCode=0 Nov 03 17:32:33.293975 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:33.293848 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" event={"ID":"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b","Type":"ContainerDied","Data":"49fb4eeeb4e85aa99e99c9a443c9aacffba382baba0a1eef04feba05fee186fe"} Nov 03 17:32:33.293975 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:33.293861 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" Nov 03 17:32:33.293975 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:33.293890 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8" event={"ID":"7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b","Type":"ContainerDied","Data":"1cf8477452758a7678e11f1dd053540b41b5edec71389db40c1db319ef6de676"} Nov 03 17:32:33.293975 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:33.293913 4947 scope.go:122] "RemoveContainer" containerID="49fb4eeeb4e85aa99e99c9a443c9aacffba382baba0a1eef04feba05fee186fe" Nov 03 17:32:33.305239 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:32:33.305209 4947 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f99db0f_b8dc_43cc_a2fd_0c3dcf5fe99b.slice\": RecentStats: unable to find data in memory cache]" Nov 03 17:32:33.306243 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:33.306220 4947 scope.go:122] "RemoveContainer" containerID="49fb4eeeb4e85aa99e99c9a443c9aacffba382baba0a1eef04feba05fee186fe" Nov 03 17:32:33.306812 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:32:33.306514 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49fb4eeeb4e85aa99e99c9a443c9aacffba382baba0a1eef04feba05fee186fe\": container with ID starting with 49fb4eeeb4e85aa99e99c9a443c9aacffba382baba0a1eef04feba05fee186fe not found: ID does not exist" containerID="49fb4eeeb4e85aa99e99c9a443c9aacffba382baba0a1eef04feba05fee186fe" Nov 03 17:32:33.306812 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:33.306547 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49fb4eeeb4e85aa99e99c9a443c9aacffba382baba0a1eef04feba05fee186fe"} err="failed to get container status \"49fb4eeeb4e85aa99e99c9a443c9aacffba382baba0a1eef04feba05fee186fe\": rpc error: code = NotFound desc = could not find container \"49fb4eeeb4e85aa99e99c9a443c9aacffba382baba0a1eef04feba05fee186fe\": container with ID starting with 49fb4eeeb4e85aa99e99c9a443c9aacffba382baba0a1eef04feba05fee186fe not found: ID does not exist" Nov 03 17:32:33.350235 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:33.350194 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8"] Nov 03 17:32:33.363554 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:33.363532 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-3a2fa2-5b4c7bffcf-ztwz8"] Nov 03 17:32:34.566822 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:32:34.566785 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b" path="/var/lib/kubelet/pods/7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b/volumes" Nov 03 17:33:00.129582 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:33:00.129491 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:33:10.462688 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:10.462649 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:33:12.760961 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.760926 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4"] Nov 03 17:33:12.761493 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761480 4947 cpu_manager.go:414] "containerMap: removing container" podUID="5af508d5-80f2-4f5a-b960-bbc2d604e97f" containerName="cleanup" Nov 03 17:33:12.761538 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761495 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af508d5-80f2-4f5a-b960-bbc2d604e97f" containerName="cleanup" Nov 03 17:33:12.761538 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761509 4947 cpu_manager.go:414] "containerMap: removing container" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" Nov 03 17:33:12.761538 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761514 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" Nov 03 17:33:12.761538 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761523 4947 cpu_manager.go:414] "containerMap: removing container" podUID="01cb2467-7a13-4f96-9c67-2cc5104d99b9" containerName="cleanup" Nov 03 17:33:12.761538 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761527 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01cb2467-7a13-4f96-9c67-2cc5104d99b9" containerName="cleanup" Nov 03 17:33:12.761538 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761537 4947 cpu_manager.go:414] "containerMap: removing container" podUID="ee442df4-03ea-4f56-9b1a-a31e84d2c861" containerName="istio-proxy" Nov 03 17:33:12.761766 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761541 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee442df4-03ea-4f56-9b1a-a31e84d2c861" containerName="istio-proxy" Nov 03 17:33:12.761766 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761551 4947 cpu_manager.go:414] "containerMap: removing container" podUID="aa47da19-7ea1-4601-bfba-18eca9086932" containerName="cleanup" Nov 03 17:33:12.761766 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761555 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa47da19-7ea1-4601-bfba-18eca9086932" containerName="cleanup" Nov 03 17:33:12.761766 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761563 4947 cpu_manager.go:414] "containerMap: removing container" podUID="7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b" containerName="maas-api" Nov 03 17:33:12.761766 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761568 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b" containerName="maas-api" Nov 03 17:33:12.761766 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761674 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af508d5-80f2-4f5a-b960-bbc2d604e97f" containerName="cleanup" Nov 03 17:33:12.761766 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761685 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa6e9f4a-d932-4a88-b640-6676db6c29f8" containerName="main" Nov 03 17:33:12.761766 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761691 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee442df4-03ea-4f56-9b1a-a31e84d2c861" containerName="istio-proxy" Nov 03 17:33:12.761766 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761718 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa47da19-7ea1-4601-bfba-18eca9086932" containerName="cleanup" Nov 03 17:33:12.761766 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761724 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f99db0f-b8dc-43cc-a2fd-0c3dcf5fe99b" containerName="maas-api" Nov 03 17:33:12.761766 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761735 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a74f616-c592-4caa-9918-b774a56814af" containerName="istio-proxy" Nov 03 17:33:12.761766 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761744 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c92bab9-7a09-4cc8-b4a9-51bd85bfd818" containerName="cleanup" Nov 03 17:33:12.761766 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761755 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba72dc1-f7d9-4cb5-ae6a-3c20337f3d3e" containerName="istio-proxy" Nov 03 17:33:12.761766 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.761764 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9644e7c0-4d23-498e-8a40-e76349d2e468" containerName="cleanup" Nov 03 17:33:12.765236 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.765220 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.781807 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.781788 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-e2d835-openshift-default-dockercfg-8f5zz\"" Nov 03 17:33:12.798687 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.798658 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4"] Nov 03 17:33:12.882230 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.882203 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-credential-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.882387 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.882244 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-data\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.882387 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.882323 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-podinfo\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.882387 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.882360 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-envoy\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.882515 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.882488 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-workload-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.882558 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.882520 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istiod-ca-cert\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.882558 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.882550 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-workload-certs\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.882642 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.882595 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-token\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.882686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.882643 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m2hh\" (UniqueName: \"kubernetes.io/projected/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-kube-api-access-4m2hh\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.984262 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.984233 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-podinfo\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.984408 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.984271 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-envoy\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.984408 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.984348 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-workload-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.984408 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.984383 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istiod-ca-cert\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.984536 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.984410 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-workload-certs\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.984536 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.984427 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-token\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.984536 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.984472 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m2hh\" (UniqueName: \"kubernetes.io/projected/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-kube-api-access-4m2hh\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.984666 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.984541 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-credential-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.984666 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.984576 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-data\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.984836 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.984815 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-workload-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.984970 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.984947 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-credential-socket\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.985189 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.985094 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-workload-certs\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.985408 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.985260 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-data\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.985408 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.985313 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istiod-ca-cert\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.986877 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.986857 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-envoy\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:12.986982 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:12.986971 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-podinfo\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:13.009421 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:13.009393 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-token\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:13.009658 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:13.009640 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m2hh\" (UniqueName: \"kubernetes.io/projected/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-kube-api-access-4m2hh\") pod \"e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:13.078650 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:13.078601 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:13.394810 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:33:13.394777 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e42485e_7fc4_4a72_ae33_cf3e4f2acb86.slice/crio-d3da91d561ea3df9504090787cc2eb47faaa6bbb1eca5906dfa02682ddccf9b7 WatchSource:0}: Error finding container d3da91d561ea3df9504090787cc2eb47faaa6bbb1eca5906dfa02682ddccf9b7: Status 404 returned error can't find the container with id d3da91d561ea3df9504090787cc2eb47faaa6bbb1eca5906dfa02682ddccf9b7 Nov 03 17:33:13.396593 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:13.396568 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4"] Nov 03 17:33:13.397309 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:13.397272 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:33:13.397419 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:13.397352 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:33:13.397419 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:13.397399 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:33:13.574572 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:13.574535 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" event={"ID":"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86","Type":"ContainerStarted","Data":"7c1360f18cf9a93c9bf64d8ce4970168d9f9c88b99c4bafb4a0ed28e0b65b818"} Nov 03 17:33:13.574775 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:13.574590 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" event={"ID":"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86","Type":"ContainerStarted","Data":"d3da91d561ea3df9504090787cc2eb47faaa6bbb1eca5906dfa02682ddccf9b7"} Nov 03 17:33:13.626123 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:13.626064 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" podStartSLOduration=1.6260509779999999 podStartE2EDuration="1.626050978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:13.623271541 +0000 UTC m=+4737.212271509" watchObservedRunningTime="2026-11-03 17:33:13.626050978 +0000 UTC m=+4737.215050944" Nov 03 17:33:14.080901 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:14.080867 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:14.082787 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:14.082755 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body= Nov 03 17:33:14.083017 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:14.082987 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" podUID="3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" Nov 03 17:33:15.079474 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:15.079438 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body= Nov 03 17:33:15.079657 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:15.079499 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" podUID="3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" Nov 03 17:33:16.079820 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:16.079785 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body= Nov 03 17:33:16.080317 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:16.079845 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" podUID="3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" Nov 03 17:33:17.079530 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:17.079484 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body= Nov 03 17:33:17.079798 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:17.079549 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" podUID="3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" Nov 03 17:33:18.078978 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:18.078943 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body= Nov 03 17:33:18.079381 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:18.078994 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" podUID="3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" Nov 03 17:33:19.079120 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:19.079088 4947 patch_prober.go:29] interesting pod/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body= Nov 03 17:33:19.079535 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:19.079141 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" podUID="3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" Nov 03 17:33:20.084244 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:20.084211 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:20.084676 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:20.084641 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:20.085359 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:20.085339 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:33:21.317099 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.317065 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg"] Nov 03 17:33:21.323425 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.323401 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.355294 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.355254 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-a9426b-openshift-default-dockercfg-tv556\"" Nov 03 17:33:21.358649 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.358621 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg"] Nov 03 17:33:21.422425 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.422393 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istiod-ca-cert\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.422629 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.422532 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-envoy\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.422629 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.422575 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-podinfo\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.422925 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.422681 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-data\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.422925 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.422815 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6v2l\" (UniqueName: \"kubernetes.io/projected/783635c1-dfe9-4471-851a-e0b3c8ee21c5-kube-api-access-d6v2l\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.422925 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.422858 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-workload-socket\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.423089 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.422972 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-token\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.423089 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.423003 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-credential-socket\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.423089 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.423045 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-workload-certs\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.524072 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.524037 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-envoy\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.524072 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.524072 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-podinfo\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.524344 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.524118 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-data\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.524344 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.524154 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6v2l\" (UniqueName: \"kubernetes.io/projected/783635c1-dfe9-4471-851a-e0b3c8ee21c5-kube-api-access-d6v2l\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.524344 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.524176 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-workload-socket\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.524344 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.524210 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-token\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.524344 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.524233 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-credential-socket\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.524344 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.524271 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-workload-certs\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.524686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.524422 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istiod-ca-cert\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.524686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.524599 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-data\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.524686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.524661 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-workload-socket\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.524861 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.524742 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-workload-certs\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.524861 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.524814 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-credential-socket\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.525111 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.525093 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istiod-ca-cert\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.527125 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.527105 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-envoy\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.527237 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.527220 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-podinfo\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.549392 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.549358 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-token\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.549544 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.549425 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6v2l\" (UniqueName: \"kubernetes.io/projected/783635c1-dfe9-4471-851a-e0b3c8ee21c5-kube-api-access-d6v2l\") pod \"e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.637849 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.637762 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:21.967110 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.967032 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg"] Nov 03 17:33:21.971554 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:33:21.971525 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783635c1_dfe9_4471_851a_e0b3c8ee21c5.slice/crio-53b6e75c653ad26c7374d60b3b0fcbe81e054d9659117c28f351b6adc439f391 WatchSource:0}: Error finding container 53b6e75c653ad26c7374d60b3b0fcbe81e054d9659117c28f351b6adc439f391: Status 404 returned error can't find the container with id 53b6e75c653ad26c7374d60b3b0fcbe81e054d9659117c28f351b6adc439f391 Nov 03 17:33:21.974592 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.974555 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:33:21.974687 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.974626 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:33:21.974687 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:21.974655 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:33:22.630149 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:22.630109 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" event={"ID":"783635c1-dfe9-4471-851a-e0b3c8ee21c5","Type":"ContainerStarted","Data":"b22f11a6e8d44d35203f747de62c8e1cfbbbb28c18859f321c109003f94278d9"} Nov 03 17:33:22.630597 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:22.630174 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" event={"ID":"783635c1-dfe9-4471-851a-e0b3c8ee21c5","Type":"ContainerStarted","Data":"53b6e75c653ad26c7374d60b3b0fcbe81e054d9659117c28f351b6adc439f391"} Nov 03 17:33:22.638427 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:22.638400 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:22.693144 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:22.693090 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" podStartSLOduration=1.69307556 podStartE2EDuration="1.69307556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:22.691114347 +0000 UTC m=+4746.280114316" watchObservedRunningTime="2026-11-03 17:33:22.69307556 +0000 UTC m=+4746.282075528" Nov 03 17:33:23.639590 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:23.639559 4947 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Nov 03 17:33:23.640040 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:23.639621 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" podUID="783635c1-dfe9-4471-851a-e0b3c8ee21c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Nov 03 17:33:24.639067 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:24.639037 4947 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Nov 03 17:33:24.639274 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:24.639098 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" podUID="783635c1-dfe9-4471-851a-e0b3c8ee21c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Nov 03 17:33:25.638440 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:25.638403 4947 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Nov 03 17:33:25.639016 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:25.638507 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" podUID="783635c1-dfe9-4471-851a-e0b3c8ee21c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Nov 03 17:33:26.639141 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:26.639107 4947 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Nov 03 17:33:26.639564 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:26.639162 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" podUID="783635c1-dfe9-4471-851a-e0b3c8ee21c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Nov 03 17:33:27.638753 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:27.638721 4947 patch_prober.go:29] interesting pod/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" start-of-body= Nov 03 17:33:27.638937 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:27.638782 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" podUID="783635c1-dfe9-4471-851a-e0b3c8ee21c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.26:15021/healthz/ready\": dial tcp 10.132.1.26:15021: connect: connection refused" Nov 03 17:33:29.034903 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:29.034868 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:29.035364 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:29.035335 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:29.036288 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:29.036270 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:33:30.132620 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:33:30.132578 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:33:46.662354 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:33:46.662312 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:34:00.129912 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:34:00.129875 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:34:12.974102 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:34:12.974069 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:34:17.359336 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:34:17.359308 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:34:17.365044 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:34:17.365021 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:34:30.130000 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:34:30.129914 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:34:48.911388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:34:48.911355 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:34:50.554709 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:34:50.554647 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8"] Nov 03 17:34:57.024285 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:34:57.024251 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k"] Nov 03 17:35:00.129101 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:35:00.129060 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:35:02.024641 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:02.024584 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" podUID="8d41dd69-ceef-4cd2-90a3-099d4562d32f" containerName="payload-processing" containerID="cri-o://b03c2f46f85885b4fe09d2dbb791e8196cd016a7aeb8fd02d86d0c8eb3e79c5b" gracePeriod=25 Nov 03 17:35:05.577177 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:05.577132 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" podUID="b48ab6d4-f151-459b-84a6-fede7e0e6d9c" containerName="main" containerID="cri-o://0cc0bfa0ddfb837f992d758c3d97b53626e48a6b9f97e48a4e9c7e0090c50989" gracePeriod=45 Nov 03 17:35:05.848594 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:05.848569 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:35:05.953542 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:05.953510 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-model-cache\") pod \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " Nov 03 17:35:05.953762 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:05.953569 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-dshm\") pod \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " Nov 03 17:35:05.953762 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:05.953621 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-tls-certs\" (UniqueName: \"kubernetes.io/secret/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-tls-certs\") pod \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " Nov 03 17:35:05.953762 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:05.953648 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-home\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-home\") pod \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " Nov 03 17:35:05.954242 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:05.953775 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-tmp-dir\") pod \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " Nov 03 17:35:05.954242 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:05.953834 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-kube-api-access-dqmtd\" (UniqueName: \"kubernetes.io/projected/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-kube-api-access-dqmtd\") pod \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\" (UID: \"b48ab6d4-f151-459b-84a6-fede7e0e6d9c\") " Nov 03 17:35:05.954242 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:05.953854 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-model-cache" pod "b48ab6d4-f151-459b-84a6-fede7e0e6d9c" (UID: "b48ab6d4-f151-459b-84a6-fede7e0e6d9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:05.954242 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:05.953949 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-home" pod "b48ab6d4-f151-459b-84a6-fede7e0e6d9c" (UID: "b48ab6d4-f151-459b-84a6-fede7e0e6d9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:05.954447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:05.954397 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:35:05.954447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:05.954419 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:35:05.954686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:05.954643 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-tmp-dir" pod "b48ab6d4-f151-459b-84a6-fede7e0e6d9c" (UID: "b48ab6d4-f151-459b-84a6-fede7e0e6d9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:05.956303 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:05.956282 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-kube-api-access-dqmtd" pod "b48ab6d4-f151-459b-84a6-fede7e0e6d9c" (UID: "b48ab6d4-f151-459b-84a6-fede7e0e6d9c"). InnerVolumeSpecName "kube-api-access-dqmtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:05.956426 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:05.956409 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-dshm" pod "b48ab6d4-f151-459b-84a6-fede7e0e6d9c" (UID: "b48ab6d4-f151-459b-84a6-fede7e0e6d9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:35:05.956490 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:05.956424 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-tls-certs" pod "b48ab6d4-f151-459b-84a6-fede7e0e6d9c" (UID: "b48ab6d4-f151-459b-84a6-fede7e0e6d9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:35:06.055881 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:06.055859 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:35:06.055980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:06.055900 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:35:06.055980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:06.055910 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqmtd\" (UniqueName: \"kubernetes.io/projected/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-kube-api-access-dqmtd\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:35:06.055980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:06.055918 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b48ab6d4-f151-459b-84a6-fede7e0e6d9c-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:35:06.275106 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:06.275059 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" podUID="8d41dd69-ceef-4cd2-90a3-099d4562d32f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:35:06.295397 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:06.295363 4947 generic.go:356] "Generic (PLEG): container finished" podID="b48ab6d4-f151-459b-84a6-fede7e0e6d9c" containerID="0cc0bfa0ddfb837f992d758c3d97b53626e48a6b9f97e48a4e9c7e0090c50989" exitCode=0 Nov 03 17:35:06.295547 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:06.295444 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" event={"ID":"b48ab6d4-f151-459b-84a6-fede7e0e6d9c","Type":"ContainerDied","Data":"0cc0bfa0ddfb837f992d758c3d97b53626e48a6b9f97e48a4e9c7e0090c50989"} Nov 03 17:35:06.295547 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:06.295462 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" Nov 03 17:35:06.295547 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:06.295473 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8" event={"ID":"b48ab6d4-f151-459b-84a6-fede7e0e6d9c","Type":"ContainerDied","Data":"7062b7d6661f4a405f8c8f9ae08f8fb99577479fce3f75f10257a5c34bb164d5"} Nov 03 17:35:06.295547 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:06.295495 4947 scope.go:122] "RemoveContainer" containerID="0cc0bfa0ddfb837f992d758c3d97b53626e48a6b9f97e48a4e9c7e0090c50989" Nov 03 17:35:06.306939 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:06.306920 4947 scope.go:122] "RemoveContainer" containerID="0cc0bfa0ddfb837f992d758c3d97b53626e48a6b9f97e48a4e9c7e0090c50989" Nov 03 17:35:06.307206 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:35:06.307186 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cc0bfa0ddfb837f992d758c3d97b53626e48a6b9f97e48a4e9c7e0090c50989\": container with ID starting with 0cc0bfa0ddfb837f992d758c3d97b53626e48a6b9f97e48a4e9c7e0090c50989 not found: ID does not exist" containerID="0cc0bfa0ddfb837f992d758c3d97b53626e48a6b9f97e48a4e9c7e0090c50989" Nov 03 17:35:06.307277 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:06.307216 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc0bfa0ddfb837f992d758c3d97b53626e48a6b9f97e48a4e9c7e0090c50989"} err="failed to get container status \"0cc0bfa0ddfb837f992d758c3d97b53626e48a6b9f97e48a4e9c7e0090c50989\": rpc error: code = NotFound desc = could not find container \"0cc0bfa0ddfb837f992d758c3d97b53626e48a6b9f97e48a4e9c7e0090c50989\": container with ID starting with 0cc0bfa0ddfb837f992d758c3d97b53626e48a6b9f97e48a4e9c7e0090c50989 not found: ID does not exist" Nov 03 17:35:06.351025 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:06.350998 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8"] Nov 03 17:35:06.364094 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:06.364075 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-1e9d48/rate-test-model-1e9d48-kserve-979559c77-z5mf8"] Nov 03 17:35:06.567754 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:06.567672 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b48ab6d4-f151-459b-84a6-fede7e0e6d9c" path="/var/lib/kubelet/pods/b48ab6d4-f151-459b-84a6-fede7e0e6d9c/volumes" Nov 03 17:35:15.956630 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:15.956597 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm"] Nov 03 17:35:15.957135 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:15.956815 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" podUID="8f62b83b-5ad8-41aa-9e18-44cab99b41fd" containerName="maas-api" containerID="cri-o://2c38ccfe731250c06a13e85bc46d7459f0df6685b1dd2530eb277f760e2d42be" gracePeriod=30 Nov 03 17:35:16.259720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.259666 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" podUID="8d41dd69-ceef-4cd2-90a3-099d4562d32f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:35:16.267519 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.264596 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" Nov 03 17:35:16.342735 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.328437 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-maas-api-tls\") pod \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\" (UID: \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\") " Nov 03 17:35:16.342735 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.328525 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-openshift-service-ca\") pod \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\" (UID: \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\") " Nov 03 17:35:16.342735 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.328747 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-kube-api-access-bzlmd\" (UniqueName: \"kubernetes.io/projected/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-kube-api-access-bzlmd\") pod \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\" (UID: \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\") " Nov 03 17:35:16.342735 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.328837 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-metrics-certs\") pod \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\" (UID: \"8f62b83b-5ad8-41aa-9e18-44cab99b41fd\") " Nov 03 17:35:16.342735 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.330201 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-openshift-service-ca" pod "8f62b83b-5ad8-41aa-9e18-44cab99b41fd" (UID: "8f62b83b-5ad8-41aa-9e18-44cab99b41fd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:16.364923 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.359905 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-maas-api-tls" pod "8f62b83b-5ad8-41aa-9e18-44cab99b41fd" (UID: "8f62b83b-5ad8-41aa-9e18-44cab99b41fd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:35:16.367206 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.367140 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-kube-api-access-bzlmd" pod "8f62b83b-5ad8-41aa-9e18-44cab99b41fd" (UID: "8f62b83b-5ad8-41aa-9e18-44cab99b41fd"). InnerVolumeSpecName "kube-api-access-bzlmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:16.369992 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.369857 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-metrics-certs" pod "8f62b83b-5ad8-41aa-9e18-44cab99b41fd" (UID: "8f62b83b-5ad8-41aa-9e18-44cab99b41fd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:35:16.379906 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.379866 4947 generic.go:356] "Generic (PLEG): container finished" podID="8f62b83b-5ad8-41aa-9e18-44cab99b41fd" containerID="2c38ccfe731250c06a13e85bc46d7459f0df6685b1dd2530eb277f760e2d42be" exitCode=0 Nov 03 17:35:16.380068 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.379927 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" event={"ID":"8f62b83b-5ad8-41aa-9e18-44cab99b41fd","Type":"ContainerDied","Data":"2c38ccfe731250c06a13e85bc46d7459f0df6685b1dd2530eb277f760e2d42be"} Nov 03 17:35:16.380068 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.379976 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" event={"ID":"8f62b83b-5ad8-41aa-9e18-44cab99b41fd","Type":"ContainerDied","Data":"a9959697fce88778dc79db527f0ba31a171d2299e4282e5e01f712f9b5817649"} Nov 03 17:35:16.380068 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.380004 4947 scope.go:122] "RemoveContainer" containerID="2c38ccfe731250c06a13e85bc46d7459f0df6685b1dd2530eb277f760e2d42be" Nov 03 17:35:16.380247 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.380143 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm" Nov 03 17:35:16.408574 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.408552 4947 scope.go:122] "RemoveContainer" containerID="2c38ccfe731250c06a13e85bc46d7459f0df6685b1dd2530eb277f760e2d42be" Nov 03 17:35:16.408928 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:35:16.408903 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c38ccfe731250c06a13e85bc46d7459f0df6685b1dd2530eb277f760e2d42be\": container with ID starting with 2c38ccfe731250c06a13e85bc46d7459f0df6685b1dd2530eb277f760e2d42be not found: ID does not exist" containerID="2c38ccfe731250c06a13e85bc46d7459f0df6685b1dd2530eb277f760e2d42be" Nov 03 17:35:16.409001 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.408940 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c38ccfe731250c06a13e85bc46d7459f0df6685b1dd2530eb277f760e2d42be"} err="failed to get container status \"2c38ccfe731250c06a13e85bc46d7459f0df6685b1dd2530eb277f760e2d42be\": rpc error: code = NotFound desc = could not find container \"2c38ccfe731250c06a13e85bc46d7459f0df6685b1dd2530eb277f760e2d42be\": container with ID starting with 2c38ccfe731250c06a13e85bc46d7459f0df6685b1dd2530eb277f760e2d42be not found: ID does not exist" Nov 03 17:35:16.438721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.436152 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzlmd\" (UniqueName: \"kubernetes.io/projected/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-kube-api-access-bzlmd\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:35:16.438721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.436192 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:35:16.438721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.436206 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:35:16.438721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.436220 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f62b83b-5ad8-41aa-9e18-44cab99b41fd-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:35:16.447846 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.446580 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm"] Nov 03 17:35:16.455522 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.455472 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d6fcae-74cf7f4dd5-zzmjm"] Nov 03 17:35:16.565418 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:16.565381 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f62b83b-5ad8-41aa-9e18-44cab99b41fd" path="/var/lib/kubelet/pods/8f62b83b-5ad8-41aa-9e18-44cab99b41fd/volumes" Nov 03 17:35:20.314375 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:20.314325 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:35:22.320237 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:22.320206 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" Nov 03 17:35:22.424050 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:22.424000 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d41dd69-ceef-4cd2-90a3-099d4562d32f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d41dd69-ceef-4cd2-90a3-099d4562d32f-plugins-config-volume\") pod \"8d41dd69-ceef-4cd2-90a3-099d4562d32f\" (UID: \"8d41dd69-ceef-4cd2-90a3-099d4562d32f\") " Nov 03 17:35:22.424292 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:22.424219 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d41dd69-ceef-4cd2-90a3-099d4562d32f-kube-api-access-vx8h5\" (UniqueName: \"kubernetes.io/projected/8d41dd69-ceef-4cd2-90a3-099d4562d32f-kube-api-access-vx8h5\") pod \"8d41dd69-ceef-4cd2-90a3-099d4562d32f\" (UID: \"8d41dd69-ceef-4cd2-90a3-099d4562d32f\") " Nov 03 17:35:22.424441 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:22.424409 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d41dd69-ceef-4cd2-90a3-099d4562d32f-plugins-config-volume" pod "8d41dd69-ceef-4cd2-90a3-099d4562d32f" (UID: "8d41dd69-ceef-4cd2-90a3-099d4562d32f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:22.424866 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:22.424835 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d41dd69-ceef-4cd2-90a3-099d4562d32f-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:35:22.426856 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:22.426830 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d41dd69-ceef-4cd2-90a3-099d4562d32f-kube-api-access-vx8h5" pod "8d41dd69-ceef-4cd2-90a3-099d4562d32f" (UID: "8d41dd69-ceef-4cd2-90a3-099d4562d32f"). InnerVolumeSpecName "kube-api-access-vx8h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:22.437445 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:22.437413 4947 generic.go:356] "Generic (PLEG): container finished" podID="8d41dd69-ceef-4cd2-90a3-099d4562d32f" containerID="b03c2f46f85885b4fe09d2dbb791e8196cd016a7aeb8fd02d86d0c8eb3e79c5b" exitCode=0 Nov 03 17:35:22.437577 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:22.437481 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" Nov 03 17:35:22.437577 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:22.437497 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" event={"ID":"8d41dd69-ceef-4cd2-90a3-099d4562d32f","Type":"ContainerDied","Data":"b03c2f46f85885b4fe09d2dbb791e8196cd016a7aeb8fd02d86d0c8eb3e79c5b"} Nov 03 17:35:22.437728 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:22.437588 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k" event={"ID":"8d41dd69-ceef-4cd2-90a3-099d4562d32f","Type":"ContainerDied","Data":"6c8aed15e13716b525e7b59796f30cdbebd09e1931f9c1cf28f11440986d5e9a"} Nov 03 17:35:22.437728 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:22.437615 4947 scope.go:122] "RemoveContainer" containerID="b03c2f46f85885b4fe09d2dbb791e8196cd016a7aeb8fd02d86d0c8eb3e79c5b" Nov 03 17:35:22.449441 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:22.449418 4947 scope.go:122] "RemoveContainer" containerID="b03c2f46f85885b4fe09d2dbb791e8196cd016a7aeb8fd02d86d0c8eb3e79c5b" Nov 03 17:35:22.449803 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:35:22.449775 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03c2f46f85885b4fe09d2dbb791e8196cd016a7aeb8fd02d86d0c8eb3e79c5b\": container with ID starting with b03c2f46f85885b4fe09d2dbb791e8196cd016a7aeb8fd02d86d0c8eb3e79c5b not found: ID does not exist" containerID="b03c2f46f85885b4fe09d2dbb791e8196cd016a7aeb8fd02d86d0c8eb3e79c5b" Nov 03 17:35:22.449882 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:22.449814 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03c2f46f85885b4fe09d2dbb791e8196cd016a7aeb8fd02d86d0c8eb3e79c5b"} err="failed to get container status \"b03c2f46f85885b4fe09d2dbb791e8196cd016a7aeb8fd02d86d0c8eb3e79c5b\": rpc error: code = NotFound desc = could not find container \"b03c2f46f85885b4fe09d2dbb791e8196cd016a7aeb8fd02d86d0c8eb3e79c5b\": container with ID starting with b03c2f46f85885b4fe09d2dbb791e8196cd016a7aeb8fd02d86d0c8eb3e79c5b not found: ID does not exist" Nov 03 17:35:22.494295 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:22.494263 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k"] Nov 03 17:35:22.500150 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:22.500125 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-da409b-5db7f6579-jxv6k"] Nov 03 17:35:22.526429 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:22.526399 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx8h5\" (UniqueName: \"kubernetes.io/projected/8d41dd69-ceef-4cd2-90a3-099d4562d32f-kube-api-access-vx8h5\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:35:22.565982 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:22.565915 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d41dd69-ceef-4cd2-90a3-099d4562d32f" path="/var/lib/kubelet/pods/8d41dd69-ceef-4cd2-90a3-099d4562d32f/volumes" Nov 03 17:35:30.155287 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:35:30.155241 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:35:44.588999 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:44.588964 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2"] Nov 03 17:35:46.592593 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:46.592560 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst"] Nov 03 17:35:46.593384 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:46.593350 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" podUID="976c6c9d-ae09-43d7-9858-89f5acbf5e20" containerName="maas-api" containerID="cri-o://971a9f9c125cf62a60fe4f7c2eb0b41158d79379941f657c579b2c48f30323d2" gracePeriod=30 Nov 03 17:35:46.907385 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:46.907360 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" Nov 03 17:35:47.116631 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.116594 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/976c6c9d-ae09-43d7-9858-89f5acbf5e20-metrics-certs\" (UniqueName: \"kubernetes.io/secret/976c6c9d-ae09-43d7-9858-89f5acbf5e20-metrics-certs\") pod \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\" (UID: \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\") " Nov 03 17:35:47.116834 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.116817 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/976c6c9d-ae09-43d7-9858-89f5acbf5e20-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/976c6c9d-ae09-43d7-9858-89f5acbf5e20-openshift-service-ca\") pod \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\" (UID: \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\") " Nov 03 17:35:47.116902 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.116874 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/976c6c9d-ae09-43d7-9858-89f5acbf5e20-kube-api-access-27zkv\" (UniqueName: \"kubernetes.io/projected/976c6c9d-ae09-43d7-9858-89f5acbf5e20-kube-api-access-27zkv\") pod \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\" (UID: \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\") " Nov 03 17:35:47.116976 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.116925 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/976c6c9d-ae09-43d7-9858-89f5acbf5e20-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/976c6c9d-ae09-43d7-9858-89f5acbf5e20-maas-api-tls\") pod \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\" (UID: \"976c6c9d-ae09-43d7-9858-89f5acbf5e20\") " Nov 03 17:35:47.117232 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.117206 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/976c6c9d-ae09-43d7-9858-89f5acbf5e20-openshift-service-ca" pod "976c6c9d-ae09-43d7-9858-89f5acbf5e20" (UID: "976c6c9d-ae09-43d7-9858-89f5acbf5e20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:35:47.118955 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.118925 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976c6c9d-ae09-43d7-9858-89f5acbf5e20-metrics-certs" pod "976c6c9d-ae09-43d7-9858-89f5acbf5e20" (UID: "976c6c9d-ae09-43d7-9858-89f5acbf5e20"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:35:47.119185 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.119168 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/976c6c9d-ae09-43d7-9858-89f5acbf5e20-maas-api-tls" pod "976c6c9d-ae09-43d7-9858-89f5acbf5e20" (UID: "976c6c9d-ae09-43d7-9858-89f5acbf5e20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:35:47.119185 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.119171 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/976c6c9d-ae09-43d7-9858-89f5acbf5e20-kube-api-access-27zkv" pod "976c6c9d-ae09-43d7-9858-89f5acbf5e20" (UID: "976c6c9d-ae09-43d7-9858-89f5acbf5e20"). InnerVolumeSpecName "kube-api-access-27zkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:35:47.218692 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.218661 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/976c6c9d-ae09-43d7-9858-89f5acbf5e20-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:35:47.218692 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.218688 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27zkv\" (UniqueName: \"kubernetes.io/projected/976c6c9d-ae09-43d7-9858-89f5acbf5e20-kube-api-access-27zkv\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:35:47.218892 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.218729 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/976c6c9d-ae09-43d7-9858-89f5acbf5e20-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:35:47.218892 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.218739 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/976c6c9d-ae09-43d7-9858-89f5acbf5e20-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:35:47.615149 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.615111 4947 generic.go:356] "Generic (PLEG): container finished" podID="976c6c9d-ae09-43d7-9858-89f5acbf5e20" containerID="971a9f9c125cf62a60fe4f7c2eb0b41158d79379941f657c579b2c48f30323d2" exitCode=0 Nov 03 17:35:47.615550 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.615192 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" Nov 03 17:35:47.615550 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.615202 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" event={"ID":"976c6c9d-ae09-43d7-9858-89f5acbf5e20","Type":"ContainerDied","Data":"971a9f9c125cf62a60fe4f7c2eb0b41158d79379941f657c579b2c48f30323d2"} Nov 03 17:35:47.615550 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.615262 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst" event={"ID":"976c6c9d-ae09-43d7-9858-89f5acbf5e20","Type":"ContainerDied","Data":"111a16d5143757ed6cc35186f6a06ad2ae0d47c70a700a8f6912cffcd18cb634"} Nov 03 17:35:47.615550 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.615290 4947 scope.go:122] "RemoveContainer" containerID="971a9f9c125cf62a60fe4f7c2eb0b41158d79379941f657c579b2c48f30323d2" Nov 03 17:35:47.625911 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.625891 4947 scope.go:122] "RemoveContainer" containerID="971a9f9c125cf62a60fe4f7c2eb0b41158d79379941f657c579b2c48f30323d2" Nov 03 17:35:47.626193 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:35:47.626174 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"971a9f9c125cf62a60fe4f7c2eb0b41158d79379941f657c579b2c48f30323d2\": container with ID starting with 971a9f9c125cf62a60fe4f7c2eb0b41158d79379941f657c579b2c48f30323d2 not found: ID does not exist" containerID="971a9f9c125cf62a60fe4f7c2eb0b41158d79379941f657c579b2c48f30323d2" Nov 03 17:35:47.626256 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.626203 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"971a9f9c125cf62a60fe4f7c2eb0b41158d79379941f657c579b2c48f30323d2"} err="failed to get container status \"971a9f9c125cf62a60fe4f7c2eb0b41158d79379941f657c579b2c48f30323d2\": rpc error: code = NotFound desc = could not find container \"971a9f9c125cf62a60fe4f7c2eb0b41158d79379941f657c579b2c48f30323d2\": container with ID starting with 971a9f9c125cf62a60fe4f7c2eb0b41158d79379941f657c579b2c48f30323d2 not found: ID does not exist" Nov 03 17:35:47.671219 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.671185 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst"] Nov 03 17:35:47.684797 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:47.684767 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5a5ab1-5464856bc9-n4dst"] Nov 03 17:35:48.565778 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:48.565748 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="976c6c9d-ae09-43d7-9858-89f5acbf5e20" path="/var/lib/kubelet/pods/976c6c9d-ae09-43d7-9858-89f5acbf5e20/volumes" Nov 03 17:35:49.590224 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:49.590177 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" podUID="9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b" containerName="payload-pre-processing" containerID="cri-o://f538fd06f43544a44679e536c9878757c3f599ff9011f023b4f12b42e3df64d7" gracePeriod=25 Nov 03 17:35:50.780521 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:50.780487 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:35:53.663927 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:53.663887 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" podUID="9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:35:56.010829 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:35:56.010797 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7"] Nov 03 17:36:00.133858 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:36:00.133766 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:36:01.011783 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:01.011735 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" podUID="95d28dd4-0aae-4f71-9bbb-24e102b03b75" containerName="payload-processing" containerID="cri-o://a83593cc161f0eb6f9b58d80bf9d0b9a6bd8f90edfd86517ade85c80badc1c67" gracePeriod=25 Nov 03 17:36:02.955629 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:02.955592 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6"] Nov 03 17:36:03.663776 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:03.663738 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" podUID="9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:07.956495 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:07.956449 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" podUID="375a1b54-34a6-4701-b10c-dcf287987ba5" containerName="payload-pre-processing" containerID="cri-o://c2f47dff1df624818e3dc60f41e5e5ed555bf91037a3c4d1e21986962737975b" gracePeriod=25 Nov 03 17:36:09.750370 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:09.750335 4947 generic.go:356] "Generic (PLEG): container finished" podID="9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b" containerID="f538fd06f43544a44679e536c9878757c3f599ff9011f023b4f12b42e3df64d7" exitCode=0 Nov 03 17:36:09.750793 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:09.750425 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" event={"ID":"9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b","Type":"ContainerDied","Data":"f538fd06f43544a44679e536c9878757c3f599ff9011f023b4f12b42e3df64d7"} Nov 03 17:36:09.845944 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:09.845880 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" podUID="95d28dd4-0aae-4f71-9bbb-24e102b03b75" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:09.867987 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:09.867962 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" Nov 03 17:36:10.036264 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:10.036224 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b-kube-api-access-rtzzs\" (UniqueName: \"kubernetes.io/projected/9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b-kube-api-access-rtzzs\") pod \"9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b\" (UID: \"9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b\") " Nov 03 17:36:10.036450 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:10.036362 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b-plugins-config-volume\") pod \"9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b\" (UID: \"9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b\") " Nov 03 17:36:10.036763 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:10.036733 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b-plugins-config-volume" pod "9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b" (UID: "9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:10.036922 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:10.036908 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:10.038658 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:10.038635 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b-kube-api-access-rtzzs" pod "9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b" (UID: "9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b"). InnerVolumeSpecName "kube-api-access-rtzzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:10.138298 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:10.138188 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtzzs\" (UniqueName: \"kubernetes.io/projected/9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b-kube-api-access-rtzzs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:10.779520 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:10.779482 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" event={"ID":"9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b","Type":"ContainerDied","Data":"da8e5efb402260a3014af6d58e813d374ff7c3e16dfe6030b1a38d1f29b0d0f9"} Nov 03 17:36:10.780108 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:10.780078 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2" Nov 03 17:36:10.780218 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:10.780142 4947 scope.go:122] "RemoveContainer" containerID="f538fd06f43544a44679e536c9878757c3f599ff9011f023b4f12b42e3df64d7" Nov 03 17:36:10.840142 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:10.840062 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2"] Nov 03 17:36:10.852915 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:10.852883 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5a5ab1-cf8797d8c-4rns2"] Nov 03 17:36:11.283356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:11.283320 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz"] Nov 03 17:36:11.316850 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:11.316818 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx"] Nov 03 17:36:12.565604 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:12.565571 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b" path="/var/lib/kubelet/pods/9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b/volumes" Nov 03 17:36:14.534421 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:14.534382 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" podUID="375a1b54-34a6-4701-b10c-dcf287987ba5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:16.284017 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:16.283976 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" podUID="7bf7a98e-51f6-4ad2-b400-4715880c82d7" containerName="payload-processing" containerID="cri-o://e010114e224003eff5c4846ff124b837287118082aa5aa7710d03fe7a01d92da" gracePeriod=25 Nov 03 17:36:16.317638 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:16.317605 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" podUID="786125e1-80ab-40fc-992d-57ddef757ddd" containerName="payload-pre-processing" containerID="cri-o://47a720a2568e251dc2e878bdf0f4bc326bbd2cd33d75785d5dc6a5076dd0f14d" gracePeriod=25 Nov 03 17:36:17.081905 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:17.081872 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg"] Nov 03 17:36:17.082098 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:17.082079 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" podUID="fa74cda3-6b8f-4553-ade8-0b4859783708" containerName="istio-proxy" containerID="cri-o://86f2e03f8ef700e19ce127a20158768d266e7d56511763ff3bb5b56498741009" gracePeriod=30 Nov 03 17:36:19.669775 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:19.669737 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:36:19.846608 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:19.846569 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" podUID="95d28dd4-0aae-4f71-9bbb-24e102b03b75" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:20.513935 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:20.513896 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr"] Nov 03 17:36:21.340994 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:21.340962 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" Nov 03 17:36:21.368711 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:21.368670 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d28dd4-0aae-4f71-9bbb-24e102b03b75-kube-api-access-ftgbf\" (UniqueName: \"kubernetes.io/projected/95d28dd4-0aae-4f71-9bbb-24e102b03b75-kube-api-access-ftgbf\") pod \"95d28dd4-0aae-4f71-9bbb-24e102b03b75\" (UID: \"95d28dd4-0aae-4f71-9bbb-24e102b03b75\") " Nov 03 17:36:21.368986 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:21.368752 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95d28dd4-0aae-4f71-9bbb-24e102b03b75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d28dd4-0aae-4f71-9bbb-24e102b03b75-plugins-config-volume\") pod \"95d28dd4-0aae-4f71-9bbb-24e102b03b75\" (UID: \"95d28dd4-0aae-4f71-9bbb-24e102b03b75\") " Nov 03 17:36:21.369255 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:21.369221 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d28dd4-0aae-4f71-9bbb-24e102b03b75-plugins-config-volume" pod "95d28dd4-0aae-4f71-9bbb-24e102b03b75" (UID: "95d28dd4-0aae-4f71-9bbb-24e102b03b75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:21.371119 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:21.371095 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d28dd4-0aae-4f71-9bbb-24e102b03b75-kube-api-access-ftgbf" pod "95d28dd4-0aae-4f71-9bbb-24e102b03b75" (UID: "95d28dd4-0aae-4f71-9bbb-24e102b03b75"). InnerVolumeSpecName "kube-api-access-ftgbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:21.470120 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:21.470081 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftgbf\" (UniqueName: \"kubernetes.io/projected/95d28dd4-0aae-4f71-9bbb-24e102b03b75-kube-api-access-ftgbf\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:21.470120 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:21.470111 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d28dd4-0aae-4f71-9bbb-24e102b03b75-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:21.890832 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:21.890793 4947 generic.go:356] "Generic (PLEG): container finished" podID="95d28dd4-0aae-4f71-9bbb-24e102b03b75" containerID="a83593cc161f0eb6f9b58d80bf9d0b9a6bd8f90edfd86517ade85c80badc1c67" exitCode=0 Nov 03 17:36:21.891049 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:21.890864 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" Nov 03 17:36:21.891049 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:21.890875 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" event={"ID":"95d28dd4-0aae-4f71-9bbb-24e102b03b75","Type":"ContainerDied","Data":"a83593cc161f0eb6f9b58d80bf9d0b9a6bd8f90edfd86517ade85c80badc1c67"} Nov 03 17:36:21.891049 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:21.890968 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7" event={"ID":"95d28dd4-0aae-4f71-9bbb-24e102b03b75","Type":"ContainerDied","Data":"72162371a88788fbd415584ed68dbf3b218079315b52efd1094d1e25c247c349"} Nov 03 17:36:21.891049 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:21.890997 4947 scope.go:122] "RemoveContainer" containerID="a83593cc161f0eb6f9b58d80bf9d0b9a6bd8f90edfd86517ade85c80badc1c67" Nov 03 17:36:21.904394 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:21.904364 4947 scope.go:122] "RemoveContainer" containerID="a83593cc161f0eb6f9b58d80bf9d0b9a6bd8f90edfd86517ade85c80badc1c67" Nov 03 17:36:21.904777 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:36:21.904752 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a83593cc161f0eb6f9b58d80bf9d0b9a6bd8f90edfd86517ade85c80badc1c67\": container with ID starting with a83593cc161f0eb6f9b58d80bf9d0b9a6bd8f90edfd86517ade85c80badc1c67 not found: ID does not exist" containerID="a83593cc161f0eb6f9b58d80bf9d0b9a6bd8f90edfd86517ade85c80badc1c67" Nov 03 17:36:21.904894 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:21.904785 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a83593cc161f0eb6f9b58d80bf9d0b9a6bd8f90edfd86517ade85c80badc1c67"} err="failed to get container status \"a83593cc161f0eb6f9b58d80bf9d0b9a6bd8f90edfd86517ade85c80badc1c67\": rpc error: code = NotFound desc = could not find container \"a83593cc161f0eb6f9b58d80bf9d0b9a6bd8f90edfd86517ade85c80badc1c67\": container with ID starting with a83593cc161f0eb6f9b58d80bf9d0b9a6bd8f90edfd86517ade85c80badc1c67 not found: ID does not exist" Nov 03 17:36:21.947427 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:21.947401 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7"] Nov 03 17:36:21.953850 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:21.953810 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3b6f35-767d667f87-2vhs7"] Nov 03 17:36:22.341186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.341161 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 17:36:22.383175 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.383134 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-podinfo\") pod \"fa74cda3-6b8f-4553-ade8-0b4859783708\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " Nov 03 17:36:22.384228 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.384202 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-credential-socket\") pod \"fa74cda3-6b8f-4553-ade8-0b4859783708\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " Nov 03 17:36:22.384396 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.384280 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-token\" (UniqueName: \"kubernetes.io/projected/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-token\") pod \"fa74cda3-6b8f-4553-ade8-0b4859783708\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " Nov 03 17:36:22.384396 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.384375 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa74cda3-6b8f-4553-ade8-0b4859783708-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa74cda3-6b8f-4553-ade8-0b4859783708-istiod-ca-cert\") pod \"fa74cda3-6b8f-4553-ade8-0b4859783708\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " Nov 03 17:36:22.386305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.384420 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-workload-socket\") pod \"fa74cda3-6b8f-4553-ade8-0b4859783708\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " Nov 03 17:36:22.386305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.384510 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-data\") pod \"fa74cda3-6b8f-4553-ade8-0b4859783708\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " Nov 03 17:36:22.386305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.384557 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-workload-certs\") pod \"fa74cda3-6b8f-4553-ade8-0b4859783708\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " Nov 03 17:36:22.386305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.384580 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-credential-socket" pod "fa74cda3-6b8f-4553-ade8-0b4859783708" (UID: "fa74cda3-6b8f-4553-ade8-0b4859783708"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:22.386305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.384594 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa74cda3-6b8f-4553-ade8-0b4859783708-kube-api-access-ghn74\" (UniqueName: \"kubernetes.io/projected/fa74cda3-6b8f-4553-ade8-0b4859783708-kube-api-access-ghn74\") pod \"fa74cda3-6b8f-4553-ade8-0b4859783708\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " Nov 03 17:36:22.386305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.384713 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-envoy\") pod \"fa74cda3-6b8f-4553-ade8-0b4859783708\" (UID: \"fa74cda3-6b8f-4553-ade8-0b4859783708\") " Nov 03 17:36:22.386305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.385018 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-data" pod "fa74cda3-6b8f-4553-ade8-0b4859783708" (UID: "fa74cda3-6b8f-4553-ade8-0b4859783708"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:22.386305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.385096 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-workload-socket" pod "fa74cda3-6b8f-4553-ade8-0b4859783708" (UID: "fa74cda3-6b8f-4553-ade8-0b4859783708"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:22.386305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.385781 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:22.386305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.385805 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:22.386305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.385820 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:22.387239 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.387167 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-workload-certs" pod "fa74cda3-6b8f-4553-ade8-0b4859783708" (UID: "fa74cda3-6b8f-4553-ade8-0b4859783708"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:22.387425 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.387401 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa74cda3-6b8f-4553-ade8-0b4859783708-istiod-ca-cert" pod "fa74cda3-6b8f-4553-ade8-0b4859783708" (UID: "fa74cda3-6b8f-4553-ade8-0b4859783708"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:22.390663 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.390636 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa74cda3-6b8f-4553-ade8-0b4859783708-kube-api-access-ghn74" pod "fa74cda3-6b8f-4553-ade8-0b4859783708" (UID: "fa74cda3-6b8f-4553-ade8-0b4859783708"). InnerVolumeSpecName "kube-api-access-ghn74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:22.397592 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.396966 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-podinfo" pod "fa74cda3-6b8f-4553-ade8-0b4859783708" (UID: "fa74cda3-6b8f-4553-ade8-0b4859783708"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:36:22.399747 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.399069 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-envoy" pod "fa74cda3-6b8f-4553-ade8-0b4859783708" (UID: "fa74cda3-6b8f-4553-ade8-0b4859783708"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:22.399747 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.399170 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-token" pod "fa74cda3-6b8f-4553-ade8-0b4859783708" (UID: "fa74cda3-6b8f-4553-ade8-0b4859783708"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:22.487421 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.487328 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:22.487421 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.487363 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghn74\" (UniqueName: \"kubernetes.io/projected/fa74cda3-6b8f-4553-ade8-0b4859783708-kube-api-access-ghn74\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:22.487421 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.487376 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:22.487421 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.487389 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:22.487421 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.487401 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa74cda3-6b8f-4553-ade8-0b4859783708-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:22.487421 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.487412 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa74cda3-6b8f-4553-ade8-0b4859783708-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:22.566108 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.566068 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d28dd4-0aae-4f71-9bbb-24e102b03b75" path="/var/lib/kubelet/pods/95d28dd4-0aae-4f71-9bbb-24e102b03b75/volumes" Nov 03 17:36:22.566610 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.566595 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z"] Nov 03 17:36:22.586532 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.586503 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4"] Nov 03 17:36:22.899870 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.899835 4947 generic.go:356] "Generic (PLEG): container finished" podID="fa74cda3-6b8f-4553-ade8-0b4859783708" containerID="86f2e03f8ef700e19ce127a20158768d266e7d56511763ff3bb5b56498741009" exitCode=0 Nov 03 17:36:22.900054 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.899883 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" Nov 03 17:36:22.900054 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.899914 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" event={"ID":"fa74cda3-6b8f-4553-ade8-0b4859783708","Type":"ContainerDied","Data":"86f2e03f8ef700e19ce127a20158768d266e7d56511763ff3bb5b56498741009"} Nov 03 17:36:22.900054 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.899969 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg" event={"ID":"fa74cda3-6b8f-4553-ade8-0b4859783708","Type":"ContainerDied","Data":"09ade277e0ce14367fe8b9120ac1371e259222c031529c9ed4a20e1693431a8d"} Nov 03 17:36:22.900054 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.899998 4947 scope.go:122] "RemoveContainer" containerID="86f2e03f8ef700e19ce127a20158768d266e7d56511763ff3bb5b56498741009" Nov 03 17:36:22.909801 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.909778 4947 scope.go:122] "RemoveContainer" containerID="11b04b70a269c86cb0a8ab3516d89e0af3d165dac4e8762ffa7de1f18c2324b9" Nov 03 17:36:22.920615 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.920593 4947 scope.go:122] "RemoveContainer" containerID="86f2e03f8ef700e19ce127a20158768d266e7d56511763ff3bb5b56498741009" Nov 03 17:36:22.920893 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:36:22.920873 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f2e03f8ef700e19ce127a20158768d266e7d56511763ff3bb5b56498741009\": container with ID starting with 86f2e03f8ef700e19ce127a20158768d266e7d56511763ff3bb5b56498741009 not found: ID does not exist" containerID="86f2e03f8ef700e19ce127a20158768d266e7d56511763ff3bb5b56498741009" Nov 03 17:36:22.920960 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.920902 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f2e03f8ef700e19ce127a20158768d266e7d56511763ff3bb5b56498741009"} err="failed to get container status \"86f2e03f8ef700e19ce127a20158768d266e7d56511763ff3bb5b56498741009\": rpc error: code = NotFound desc = could not find container \"86f2e03f8ef700e19ce127a20158768d266e7d56511763ff3bb5b56498741009\": container with ID starting with 86f2e03f8ef700e19ce127a20158768d266e7d56511763ff3bb5b56498741009 not found: ID does not exist" Nov 03 17:36:22.920960 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.920921 4947 scope.go:122] "RemoveContainer" containerID="11b04b70a269c86cb0a8ab3516d89e0af3d165dac4e8762ffa7de1f18c2324b9" Nov 03 17:36:22.921184 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:36:22.921167 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11b04b70a269c86cb0a8ab3516d89e0af3d165dac4e8762ffa7de1f18c2324b9\": container with ID starting with 11b04b70a269c86cb0a8ab3516d89e0af3d165dac4e8762ffa7de1f18c2324b9 not found: ID does not exist" containerID="11b04b70a269c86cb0a8ab3516d89e0af3d165dac4e8762ffa7de1f18c2324b9" Nov 03 17:36:22.921245 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.921186 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11b04b70a269c86cb0a8ab3516d89e0af3d165dac4e8762ffa7de1f18c2324b9"} err="failed to get container status \"11b04b70a269c86cb0a8ab3516d89e0af3d165dac4e8762ffa7de1f18c2324b9\": rpc error: code = NotFound desc = could not find container \"11b04b70a269c86cb0a8ab3516d89e0af3d165dac4e8762ffa7de1f18c2324b9\": container with ID starting with 11b04b70a269c86cb0a8ab3516d89e0af3d165dac4e8762ffa7de1f18c2324b9 not found: ID does not exist" Nov 03 17:36:22.959052 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.959024 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg"] Nov 03 17:36:22.973305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:22.973270 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-1e9d48-openshift-default-6d5b4cc6c-gpjfg"] Nov 03 17:36:24.193906 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:24.193864 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" podUID="7bf7a98e-51f6-4ad2-b400-4715880c82d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:24.195748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:24.195690 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" podUID="786125e1-80ab-40fc-992d-57ddef757ddd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:24.534791 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:24.534748 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" podUID="375a1b54-34a6-4701-b10c-dcf287987ba5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:24.567000 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:24.566972 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa74cda3-6b8f-4553-ade8-0b4859783708" path="/var/lib/kubelet/pods/fa74cda3-6b8f-4553-ade8-0b4859783708/volumes" Nov 03 17:36:25.514434 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:25.514386 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" podUID="fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b" containerName="payload-processing" containerID="cri-o://573ea6ebd0da011d9bc37a937f20ae11c9c19c1f18cf92109b6beaa213faf6a4" gracePeriod=25 Nov 03 17:36:28.237809 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:28.237739 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" Nov 03 17:36:28.389869 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:28.389653 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/375a1b54-34a6-4701-b10c-dcf287987ba5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/375a1b54-34a6-4701-b10c-dcf287987ba5-plugins-config-volume\") pod \"375a1b54-34a6-4701-b10c-dcf287987ba5\" (UID: \"375a1b54-34a6-4701-b10c-dcf287987ba5\") " Nov 03 17:36:28.389869 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:28.389809 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/375a1b54-34a6-4701-b10c-dcf287987ba5-kube-api-access-x6lcc\" (UniqueName: \"kubernetes.io/projected/375a1b54-34a6-4701-b10c-dcf287987ba5-kube-api-access-x6lcc\") pod \"375a1b54-34a6-4701-b10c-dcf287987ba5\" (UID: \"375a1b54-34a6-4701-b10c-dcf287987ba5\") " Nov 03 17:36:28.391726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:28.390901 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/375a1b54-34a6-4701-b10c-dcf287987ba5-plugins-config-volume" pod "375a1b54-34a6-4701-b10c-dcf287987ba5" (UID: "375a1b54-34a6-4701-b10c-dcf287987ba5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:28.395291 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:28.395252 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/375a1b54-34a6-4701-b10c-dcf287987ba5-kube-api-access-x6lcc" pod "375a1b54-34a6-4701-b10c-dcf287987ba5" (UID: "375a1b54-34a6-4701-b10c-dcf287987ba5"). InnerVolumeSpecName "kube-api-access-x6lcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:28.492036 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:28.492005 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6lcc\" (UniqueName: \"kubernetes.io/projected/375a1b54-34a6-4701-b10c-dcf287987ba5-kube-api-access-x6lcc\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:28.492036 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:28.492032 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/375a1b54-34a6-4701-b10c-dcf287987ba5-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:28.958490 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:28.958396 4947 generic.go:356] "Generic (PLEG): container finished" podID="375a1b54-34a6-4701-b10c-dcf287987ba5" containerID="c2f47dff1df624818e3dc60f41e5e5ed555bf91037a3c4d1e21986962737975b" exitCode=0 Nov 03 17:36:28.958490 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:28.958441 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" Nov 03 17:36:28.958490 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:28.958462 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" event={"ID":"375a1b54-34a6-4701-b10c-dcf287987ba5","Type":"ContainerDied","Data":"c2f47dff1df624818e3dc60f41e5e5ed555bf91037a3c4d1e21986962737975b"} Nov 03 17:36:28.958924 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:28.958516 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6" event={"ID":"375a1b54-34a6-4701-b10c-dcf287987ba5","Type":"ContainerDied","Data":"f4c03028fc903841ff50fc5c5a259afa626872bef6320b4ba4d958944f817ba8"} Nov 03 17:36:28.958924 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:28.958547 4947 scope.go:122] "RemoveContainer" containerID="c2f47dff1df624818e3dc60f41e5e5ed555bf91037a3c4d1e21986962737975b" Nov 03 17:36:28.970938 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:28.970912 4947 scope.go:122] "RemoveContainer" containerID="c2f47dff1df624818e3dc60f41e5e5ed555bf91037a3c4d1e21986962737975b" Nov 03 17:36:28.971225 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:36:28.971197 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f47dff1df624818e3dc60f41e5e5ed555bf91037a3c4d1e21986962737975b\": container with ID starting with c2f47dff1df624818e3dc60f41e5e5ed555bf91037a3c4d1e21986962737975b not found: ID does not exist" containerID="c2f47dff1df624818e3dc60f41e5e5ed555bf91037a3c4d1e21986962737975b" Nov 03 17:36:28.971311 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:28.971236 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f47dff1df624818e3dc60f41e5e5ed555bf91037a3c4d1e21986962737975b"} err="failed to get container status \"c2f47dff1df624818e3dc60f41e5e5ed555bf91037a3c4d1e21986962737975b\": rpc error: code = NotFound desc = could not find container \"c2f47dff1df624818e3dc60f41e5e5ed555bf91037a3c4d1e21986962737975b\": container with ID starting with c2f47dff1df624818e3dc60f41e5e5ed555bf91037a3c4d1e21986962737975b not found: ID does not exist" Nov 03 17:36:29.013817 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:29.013788 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6"] Nov 03 17:36:29.026799 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:29.026764 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-1e9d48-766d869bc-wzdf6"] Nov 03 17:36:30.130680 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:36:30.130635 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:36:30.566816 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:30.566772 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="375a1b54-34a6-4701-b10c-dcf287987ba5" path="/var/lib/kubelet/pods/375a1b54-34a6-4701-b10c-dcf287987ba5/volumes" Nov 03 17:36:32.957044 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:32.957014 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:36:34.191753 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:34.191683 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" podUID="786125e1-80ab-40fc-992d-57ddef757ddd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:34.192159 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:34.192072 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" podUID="7bf7a98e-51f6-4ad2-b400-4715880c82d7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:35.259894 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:35.259852 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" podUID="fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:36.642389 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:36.642353 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" Nov 03 17:36:36.759664 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:36.759630 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" Nov 03 17:36:36.810020 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:36.809990 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf7a98e-51f6-4ad2-b400-4715880c82d7-kube-api-access-tknbh\" (UniqueName: \"kubernetes.io/projected/7bf7a98e-51f6-4ad2-b400-4715880c82d7-kube-api-access-tknbh\") pod \"7bf7a98e-51f6-4ad2-b400-4715880c82d7\" (UID: \"7bf7a98e-51f6-4ad2-b400-4715880c82d7\") " Nov 03 17:36:36.810225 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:36.810149 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf7a98e-51f6-4ad2-b400-4715880c82d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf7a98e-51f6-4ad2-b400-4715880c82d7-plugins-config-volume\") pod \"7bf7a98e-51f6-4ad2-b400-4715880c82d7\" (UID: \"7bf7a98e-51f6-4ad2-b400-4715880c82d7\") " Nov 03 17:36:36.810895 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:36.810870 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf7a98e-51f6-4ad2-b400-4715880c82d7-plugins-config-volume" pod "7bf7a98e-51f6-4ad2-b400-4715880c82d7" (UID: "7bf7a98e-51f6-4ad2-b400-4715880c82d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:36.812369 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:36.812344 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf7a98e-51f6-4ad2-b400-4715880c82d7-kube-api-access-tknbh" pod "7bf7a98e-51f6-4ad2-b400-4715880c82d7" (UID: "7bf7a98e-51f6-4ad2-b400-4715880c82d7"). InnerVolumeSpecName "kube-api-access-tknbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:36.911033 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:36.910935 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/786125e1-80ab-40fc-992d-57ddef757ddd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786125e1-80ab-40fc-992d-57ddef757ddd-plugins-config-volume\") pod \"786125e1-80ab-40fc-992d-57ddef757ddd\" (UID: \"786125e1-80ab-40fc-992d-57ddef757ddd\") " Nov 03 17:36:36.911230 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:36.911088 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/786125e1-80ab-40fc-992d-57ddef757ddd-kube-api-access-tpnbp\" (UniqueName: \"kubernetes.io/projected/786125e1-80ab-40fc-992d-57ddef757ddd-kube-api-access-tpnbp\") pod \"786125e1-80ab-40fc-992d-57ddef757ddd\" (UID: \"786125e1-80ab-40fc-992d-57ddef757ddd\") " Nov 03 17:36:36.911358 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:36.911331 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/786125e1-80ab-40fc-992d-57ddef757ddd-plugins-config-volume" pod "786125e1-80ab-40fc-992d-57ddef757ddd" (UID: "786125e1-80ab-40fc-992d-57ddef757ddd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:36.911554 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:36.911537 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf7a98e-51f6-4ad2-b400-4715880c82d7-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:36.911618 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:36.911555 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tknbh\" (UniqueName: \"kubernetes.io/projected/7bf7a98e-51f6-4ad2-b400-4715880c82d7-kube-api-access-tknbh\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:36.911618 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:36.911564 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/786125e1-80ab-40fc-992d-57ddef757ddd-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:36.913859 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:36.913833 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786125e1-80ab-40fc-992d-57ddef757ddd-kube-api-access-tpnbp" pod "786125e1-80ab-40fc-992d-57ddef757ddd" (UID: "786125e1-80ab-40fc-992d-57ddef757ddd"). InnerVolumeSpecName "kube-api-access-tpnbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:37.012980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.012943 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpnbp\" (UniqueName: \"kubernetes.io/projected/786125e1-80ab-40fc-992d-57ddef757ddd-kube-api-access-tpnbp\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:37.030387 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.030353 4947 generic.go:356] "Generic (PLEG): container finished" podID="7bf7a98e-51f6-4ad2-b400-4715880c82d7" containerID="e010114e224003eff5c4846ff124b837287118082aa5aa7710d03fe7a01d92da" exitCode=0 Nov 03 17:36:37.030592 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.030417 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" Nov 03 17:36:37.030592 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.030437 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" event={"ID":"7bf7a98e-51f6-4ad2-b400-4715880c82d7","Type":"ContainerDied","Data":"e010114e224003eff5c4846ff124b837287118082aa5aa7710d03fe7a01d92da"} Nov 03 17:36:37.030592 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.030500 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz" event={"ID":"7bf7a98e-51f6-4ad2-b400-4715880c82d7","Type":"ContainerDied","Data":"1fdc37be9d72d7651ab99aacc4ff7db2f40e908a23cca6f678df4eb98af44f42"} Nov 03 17:36:37.030592 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.030530 4947 scope.go:122] "RemoveContainer" containerID="e010114e224003eff5c4846ff124b837287118082aa5aa7710d03fe7a01d92da" Nov 03 17:36:37.032594 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.032570 4947 generic.go:356] "Generic (PLEG): container finished" podID="786125e1-80ab-40fc-992d-57ddef757ddd" containerID="47a720a2568e251dc2e878bdf0f4bc326bbd2cd33d75785d5dc6a5076dd0f14d" exitCode=0 Nov 03 17:36:37.032759 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.032623 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" event={"ID":"786125e1-80ab-40fc-992d-57ddef757ddd","Type":"ContainerDied","Data":"47a720a2568e251dc2e878bdf0f4bc326bbd2cd33d75785d5dc6a5076dd0f14d"} Nov 03 17:36:37.032759 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.032638 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" Nov 03 17:36:37.032759 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.032664 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx" event={"ID":"786125e1-80ab-40fc-992d-57ddef757ddd","Type":"ContainerDied","Data":"4e6f9604594fd7e2959383a3d359fa5d2572a7f2f32fe80eaf4271934f539b18"} Nov 03 17:36:37.043821 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.043801 4947 scope.go:122] "RemoveContainer" containerID="e010114e224003eff5c4846ff124b837287118082aa5aa7710d03fe7a01d92da" Nov 03 17:36:37.044317 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:36:37.044145 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e010114e224003eff5c4846ff124b837287118082aa5aa7710d03fe7a01d92da\": container with ID starting with e010114e224003eff5c4846ff124b837287118082aa5aa7710d03fe7a01d92da not found: ID does not exist" containerID="e010114e224003eff5c4846ff124b837287118082aa5aa7710d03fe7a01d92da" Nov 03 17:36:37.044317 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.044182 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e010114e224003eff5c4846ff124b837287118082aa5aa7710d03fe7a01d92da"} err="failed to get container status \"e010114e224003eff5c4846ff124b837287118082aa5aa7710d03fe7a01d92da\": rpc error: code = NotFound desc = could not find container \"e010114e224003eff5c4846ff124b837287118082aa5aa7710d03fe7a01d92da\": container with ID starting with e010114e224003eff5c4846ff124b837287118082aa5aa7710d03fe7a01d92da not found: ID does not exist" Nov 03 17:36:37.044317 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.044219 4947 scope.go:122] "RemoveContainer" containerID="47a720a2568e251dc2e878bdf0f4bc326bbd2cd33d75785d5dc6a5076dd0f14d" Nov 03 17:36:37.053495 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.053478 4947 scope.go:122] "RemoveContainer" containerID="47a720a2568e251dc2e878bdf0f4bc326bbd2cd33d75785d5dc6a5076dd0f14d" Nov 03 17:36:37.053813 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:36:37.053793 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47a720a2568e251dc2e878bdf0f4bc326bbd2cd33d75785d5dc6a5076dd0f14d\": container with ID starting with 47a720a2568e251dc2e878bdf0f4bc326bbd2cd33d75785d5dc6a5076dd0f14d not found: ID does not exist" containerID="47a720a2568e251dc2e878bdf0f4bc326bbd2cd33d75785d5dc6a5076dd0f14d" Nov 03 17:36:37.053873 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.053821 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47a720a2568e251dc2e878bdf0f4bc326bbd2cd33d75785d5dc6a5076dd0f14d"} err="failed to get container status \"47a720a2568e251dc2e878bdf0f4bc326bbd2cd33d75785d5dc6a5076dd0f14d\": rpc error: code = NotFound desc = could not find container \"47a720a2568e251dc2e878bdf0f4bc326bbd2cd33d75785d5dc6a5076dd0f14d\": container with ID starting with 47a720a2568e251dc2e878bdf0f4bc326bbd2cd33d75785d5dc6a5076dd0f14d not found: ID does not exist" Nov 03 17:36:37.085890 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.085862 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz"] Nov 03 17:36:37.089930 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.089908 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-3a0e27-5fbccb5467-n27hz"] Nov 03 17:36:37.128149 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.128091 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx"] Nov 03 17:36:37.131973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.131948 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-3a0e27-655dbd4d6c-z8wgx"] Nov 03 17:36:37.589920 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.589829 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" containerID="cri-o://3516a3a7c217beee742963a40c8c765115762d317ce1ded8cbf73c5ef0c8e638" gracePeriod=45 Nov 03 17:36:37.608160 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.608113 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" containerID="cri-o://c2e571a588fd7326173d52cbdb513a675260c068d3199f9820a272cd07535d18" gracePeriod=45 Nov 03 17:36:37.876344 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.876317 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 17:36:37.888772 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:37.888750 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 17:36:38.026971 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.026933 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-dshm\") pod \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " Nov 03 17:36:38.027168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.026999 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-home\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-home\") pod \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " Nov 03 17:36:38.027168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.027052 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-model-cache\") pod \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " Nov 03 17:36:38.027168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.027079 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-model-cache\") pod \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " Nov 03 17:36:38.027168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.027113 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-kube-api-access-xl4td\" (UniqueName: \"kubernetes.io/projected/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-kube-api-access-xl4td\") pod \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " Nov 03 17:36:38.027168 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.027164 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-home\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-home\") pod \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " Nov 03 17:36:38.027471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.027286 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-tmp-dir\") pod \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " Nov 03 17:36:38.027471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.027323 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-home" pod "0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" (UID: "0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:38.027471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.027341 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-model-cache" pod "0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" (UID: "0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:38.027471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.027337 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-tls-certs\") pod \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " Nov 03 17:36:38.027471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.027428 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-tmp-dir\") pod \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " Nov 03 17:36:38.027959 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.027481 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-dshm\") pod \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " Nov 03 17:36:38.027959 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.027523 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-kube-api-access-qphgd\" (UniqueName: \"kubernetes.io/projected/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-kube-api-access-qphgd\") pod \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\" (UID: \"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780\") " Nov 03 17:36:38.027959 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.027648 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-tls-certs\") pod \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\" (UID: \"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd\") " Nov 03 17:36:38.032270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.029431 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.032270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.029461 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.032270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.029434 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-tmp-dir" pod "0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" (UID: "0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:38.032270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.029736 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-model-cache" pod "fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" (UID: "fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:38.032270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.030035 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-dshm" pod "fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" (UID: "fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:38.032270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.030046 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-home" pod "fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" (UID: "fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:38.032270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.030056 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-tmp-dir" pod "fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" (UID: "fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:38.032270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.032145 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-tls-certs" pod "0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" (UID: "0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:38.032270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.032175 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-kube-api-access-xl4td" pod "0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" (UID: "0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd"). InnerVolumeSpecName "kube-api-access-xl4td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:38.032270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.032203 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-kube-api-access-qphgd" pod "fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" (UID: "fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780"). InnerVolumeSpecName "kube-api-access-qphgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:38.032270 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.032210 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-tls-certs" pod "fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" (UID: "fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:38.034467 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.034440 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-dshm" pod "0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" (UID: "0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:38.042108 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.042085 4947 generic.go:356] "Generic (PLEG): container finished" podID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerID="3516a3a7c217beee742963a40c8c765115762d317ce1ded8cbf73c5ef0c8e638" exitCode=0 Nov 03 17:36:38.042230 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.042219 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" Nov 03 17:36:38.042936 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.042907 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" event={"ID":"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd","Type":"ContainerDied","Data":"3516a3a7c217beee742963a40c8c765115762d317ce1ded8cbf73c5ef0c8e638"} Nov 03 17:36:38.043089 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.043071 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z" event={"ID":"0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd","Type":"ContainerDied","Data":"7eeb586f9ee4b99e6b4a5e589d0b51cbacd1eb63a512d0e978bc6c6d24603a4b"} Nov 03 17:36:38.043226 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.043203 4947 scope.go:122] "RemoveContainer" containerID="3516a3a7c217beee742963a40c8c765115762d317ce1ded8cbf73c5ef0c8e638" Nov 03 17:36:38.044982 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.044958 4947 generic.go:356] "Generic (PLEG): container finished" podID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerID="c2e571a588fd7326173d52cbdb513a675260c068d3199f9820a272cd07535d18" exitCode=0 Nov 03 17:36:38.045088 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.045040 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" event={"ID":"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780","Type":"ContainerDied","Data":"c2e571a588fd7326173d52cbdb513a675260c068d3199f9820a272cd07535d18"} Nov 03 17:36:38.045088 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.045067 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" event={"ID":"fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780","Type":"ContainerDied","Data":"9cf6fd275beeb850a2568531834a85bf2286d6cd59b3511ad99647b5c17c9ace"} Nov 03 17:36:38.045231 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.045125 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4" Nov 03 17:36:38.063323 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.063293 4947 scope.go:122] "RemoveContainer" containerID="3516a3a7c217beee742963a40c8c765115762d317ce1ded8cbf73c5ef0c8e638" Nov 03 17:36:38.063588 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:36:38.063567 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3516a3a7c217beee742963a40c8c765115762d317ce1ded8cbf73c5ef0c8e638\": container with ID starting with 3516a3a7c217beee742963a40c8c765115762d317ce1ded8cbf73c5ef0c8e638 not found: ID does not exist" containerID="3516a3a7c217beee742963a40c8c765115762d317ce1ded8cbf73c5ef0c8e638" Nov 03 17:36:38.063671 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.063596 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3516a3a7c217beee742963a40c8c765115762d317ce1ded8cbf73c5ef0c8e638"} err="failed to get container status \"3516a3a7c217beee742963a40c8c765115762d317ce1ded8cbf73c5ef0c8e638\": rpc error: code = NotFound desc = could not find container \"3516a3a7c217beee742963a40c8c765115762d317ce1ded8cbf73c5ef0c8e638\": container with ID starting with 3516a3a7c217beee742963a40c8c765115762d317ce1ded8cbf73c5ef0c8e638 not found: ID does not exist" Nov 03 17:36:38.063671 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.063612 4947 scope.go:122] "RemoveContainer" containerID="c2e571a588fd7326173d52cbdb513a675260c068d3199f9820a272cd07535d18" Nov 03 17:36:38.072465 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.072441 4947 scope.go:122] "RemoveContainer" containerID="c2e571a588fd7326173d52cbdb513a675260c068d3199f9820a272cd07535d18" Nov 03 17:36:38.072872 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:36:38.072833 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e571a588fd7326173d52cbdb513a675260c068d3199f9820a272cd07535d18\": container with ID starting with c2e571a588fd7326173d52cbdb513a675260c068d3199f9820a272cd07535d18 not found: ID does not exist" containerID="c2e571a588fd7326173d52cbdb513a675260c068d3199f9820a272cd07535d18" Nov 03 17:36:38.072964 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.072883 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e571a588fd7326173d52cbdb513a675260c068d3199f9820a272cd07535d18"} err="failed to get container status \"c2e571a588fd7326173d52cbdb513a675260c068d3199f9820a272cd07535d18\": rpc error: code = NotFound desc = could not find container \"c2e571a588fd7326173d52cbdb513a675260c068d3199f9820a272cd07535d18\": container with ID starting with c2e571a588fd7326173d52cbdb513a675260c068d3199f9820a272cd07535d18 not found: ID does not exist" Nov 03 17:36:38.097489 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.097436 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z"] Nov 03 17:36:38.103364 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.103341 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/distinct-2-w0-71d7b4-kserve-9f8bc4597-j847z"] Nov 03 17:36:38.130308 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.130279 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.130308 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.130311 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.130540 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.130323 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.130540 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.130334 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.130540 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.130346 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qphgd\" (UniqueName: \"kubernetes.io/projected/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-kube-api-access-qphgd\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.130540 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.130359 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.130540 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.130374 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.130540 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.130386 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.130540 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.130397 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl4td\" (UniqueName: \"kubernetes.io/projected/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd-kube-api-access-xl4td\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.130540 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.130409 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:38.158626 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.158597 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4"] Nov 03 17:36:38.167149 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.167129 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-71d7b4/simulator-w0-71d7b4-kserve-cb95b595d-dwcd4"] Nov 03 17:36:38.566354 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.566323 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" path="/var/lib/kubelet/pods/0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd/volumes" Nov 03 17:36:38.566760 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.566745 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="786125e1-80ab-40fc-992d-57ddef757ddd" path="/var/lib/kubelet/pods/786125e1-80ab-40fc-992d-57ddef757ddd/volumes" Nov 03 17:36:38.567095 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.567083 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf7a98e-51f6-4ad2-b400-4715880c82d7" path="/var/lib/kubelet/pods/7bf7a98e-51f6-4ad2-b400-4715880c82d7/volumes" Nov 03 17:36:38.567800 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:38.567784 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" path="/var/lib/kubelet/pods/fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780/volumes" Nov 03 17:36:41.913462 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:41.913429 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf"] Nov 03 17:36:42.075497 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:42.075453 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r"] Nov 03 17:36:44.695615 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:44.695582 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:36:45.259381 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:45.259346 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" podUID="fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:45.834892 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:45.834861 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" Nov 03 17:36:45.918396 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:45.918340 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b-plugins-config-volume\") pod \"fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b\" (UID: \"fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b\") " Nov 03 17:36:45.918614 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:45.918475 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b-kube-api-access-hqcb4\" (UniqueName: \"kubernetes.io/projected/fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b-kube-api-access-hqcb4\") pod \"fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b\" (UID: \"fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b\") " Nov 03 17:36:45.918838 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:45.918808 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b-plugins-config-volume" pod "fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b" (UID: "fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:36:45.919002 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:45.918979 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:45.957676 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:45.957631 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b-kube-api-access-hqcb4" pod "fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b" (UID: "fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b"). InnerVolumeSpecName "kube-api-access-hqcb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:46.020226 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:46.020184 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqcb4\" (UniqueName: \"kubernetes.io/projected/fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b-kube-api-access-hqcb4\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:46.108418 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:46.108310 4947 generic.go:356] "Generic (PLEG): container finished" podID="fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b" containerID="573ea6ebd0da011d9bc37a937f20ae11c9c19c1f18cf92109b6beaa213faf6a4" exitCode=0 Nov 03 17:36:46.108418 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:46.108394 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" event={"ID":"fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b","Type":"ContainerDied","Data":"573ea6ebd0da011d9bc37a937f20ae11c9c19c1f18cf92109b6beaa213faf6a4"} Nov 03 17:36:46.108418 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:46.108396 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" Nov 03 17:36:46.108781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:46.108472 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr" event={"ID":"fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b","Type":"ContainerDied","Data":"c5fd37415aa43ed5b0a022edc921046f51a48735c612ccb10f4ac0b05060919c"} Nov 03 17:36:46.108781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:46.108495 4947 scope.go:122] "RemoveContainer" containerID="573ea6ebd0da011d9bc37a937f20ae11c9c19c1f18cf92109b6beaa213faf6a4" Nov 03 17:36:46.119944 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:46.119878 4947 scope.go:122] "RemoveContainer" containerID="573ea6ebd0da011d9bc37a937f20ae11c9c19c1f18cf92109b6beaa213faf6a4" Nov 03 17:36:46.120319 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:36:46.120248 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573ea6ebd0da011d9bc37a937f20ae11c9c19c1f18cf92109b6beaa213faf6a4\": container with ID starting with 573ea6ebd0da011d9bc37a937f20ae11c9c19c1f18cf92109b6beaa213faf6a4 not found: ID does not exist" containerID="573ea6ebd0da011d9bc37a937f20ae11c9c19c1f18cf92109b6beaa213faf6a4" Nov 03 17:36:46.120319 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:46.120303 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573ea6ebd0da011d9bc37a937f20ae11c9c19c1f18cf92109b6beaa213faf6a4"} err="failed to get container status \"573ea6ebd0da011d9bc37a937f20ae11c9c19c1f18cf92109b6beaa213faf6a4\": rpc error: code = NotFound desc = could not find container \"573ea6ebd0da011d9bc37a937f20ae11c9c19c1f18cf92109b6beaa213faf6a4\": container with ID starting with 573ea6ebd0da011d9bc37a937f20ae11c9c19c1f18cf92109b6beaa213faf6a4 not found: ID does not exist" Nov 03 17:36:46.165634 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:46.165601 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr"] Nov 03 17:36:46.178781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:46.178757 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c6f303-74bb68cc89-fmnzr"] Nov 03 17:36:46.565142 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:46.565113 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b" path="/var/lib/kubelet/pods/fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b/volumes" Nov 03 17:36:46.913980 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:46.913882 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" podUID="43b5b746-c30a-428b-95ef-7f94cc3f6fd7" containerName="payload-pre-processing" containerID="cri-o://771f33136531bc0165308faaae1d3d5be0ae19b76651ae87a894a190e35d0f1f" gracePeriod=25 Nov 03 17:36:51.562189 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:51.562152 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" podUID="43b5b746-c30a-428b-95ef-7f94cc3f6fd7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:36:57.097688 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.097645 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" containerID="cri-o://2fb53936824fca8eaa6ecd6f840d476533c01fdccedcb54acdcbb3063f83fde0" gracePeriod=45 Nov 03 17:36:57.413066 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.413039 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 17:36:57.567665 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.567622 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-tmp-dir\") pod \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " Nov 03 17:36:57.567892 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.567679 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-dshm\") pod \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " Nov 03 17:36:57.567892 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.567771 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40eb248-b0bd-43ab-821c-b21ad6f8a364-kube-api-access-2qxkz\" (UniqueName: \"kubernetes.io/projected/f40eb248-b0bd-43ab-821c-b21ad6f8a364-kube-api-access-2qxkz\") pod \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " Nov 03 17:36:57.568034 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.567914 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-home\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-home\") pod \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " Nov 03 17:36:57.568034 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.567943 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-tmp-dir" pod "f40eb248-b0bd-43ab-821c-b21ad6f8a364" (UID: "f40eb248-b0bd-43ab-821c-b21ad6f8a364"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:57.568034 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.567960 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-model-cache\") pod \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " Nov 03 17:36:57.568229 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.568129 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-home" pod "f40eb248-b0bd-43ab-821c-b21ad6f8a364" (UID: "f40eb248-b0bd-43ab-821c-b21ad6f8a364"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:57.568229 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.568131 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f40eb248-b0bd-43ab-821c-b21ad6f8a364-tls-certs\" (UniqueName: \"kubernetes.io/secret/f40eb248-b0bd-43ab-821c-b21ad6f8a364-tls-certs\") pod \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\" (UID: \"f40eb248-b0bd-43ab-821c-b21ad6f8a364\") " Nov 03 17:36:57.568474 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.568443 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-model-cache" pod "f40eb248-b0bd-43ab-821c-b21ad6f8a364" (UID: "f40eb248-b0bd-43ab-821c-b21ad6f8a364"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:57.572736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.568655 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.572736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.568678 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.572736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.568730 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.572736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.571390 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f40eb248-b0bd-43ab-821c-b21ad6f8a364-tls-certs" pod "f40eb248-b0bd-43ab-821c-b21ad6f8a364" (UID: "f40eb248-b0bd-43ab-821c-b21ad6f8a364"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:36:57.572736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.571467 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-dshm" pod "f40eb248-b0bd-43ab-821c-b21ad6f8a364" (UID: "f40eb248-b0bd-43ab-821c-b21ad6f8a364"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:36:57.572736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.572011 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40eb248-b0bd-43ab-821c-b21ad6f8a364-kube-api-access-2qxkz" pod "f40eb248-b0bd-43ab-821c-b21ad6f8a364" (UID: "f40eb248-b0bd-43ab-821c-b21ad6f8a364"). InnerVolumeSpecName "kube-api-access-2qxkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:36:57.670425 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.670322 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f40eb248-b0bd-43ab-821c-b21ad6f8a364-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.670425 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.670359 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f40eb248-b0bd-43ab-821c-b21ad6f8a364-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:57.670425 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:57.670372 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qxkz\" (UniqueName: \"kubernetes.io/projected/f40eb248-b0bd-43ab-821c-b21ad6f8a364-kube-api-access-2qxkz\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:36:58.189876 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:58.189837 4947 generic.go:356] "Generic (PLEG): container finished" podID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerID="2fb53936824fca8eaa6ecd6f840d476533c01fdccedcb54acdcbb3063f83fde0" exitCode=0 Nov 03 17:36:58.190381 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:58.189882 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" Nov 03 17:36:58.190381 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:58.189922 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" event={"ID":"f40eb248-b0bd-43ab-821c-b21ad6f8a364","Type":"ContainerDied","Data":"2fb53936824fca8eaa6ecd6f840d476533c01fdccedcb54acdcbb3063f83fde0"} Nov 03 17:36:58.190381 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:58.189967 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r" event={"ID":"f40eb248-b0bd-43ab-821c-b21ad6f8a364","Type":"ContainerDied","Data":"b415798784cf61b5a9982f88d13b3d1b4599b748ae1ceb74dfa299f1fd000c6a"} Nov 03 17:36:58.190381 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:58.189988 4947 scope.go:122] "RemoveContainer" containerID="2fb53936824fca8eaa6ecd6f840d476533c01fdccedcb54acdcbb3063f83fde0" Nov 03 17:36:58.202477 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:58.202448 4947 scope.go:122] "RemoveContainer" containerID="2fb53936824fca8eaa6ecd6f840d476533c01fdccedcb54acdcbb3063f83fde0" Nov 03 17:36:58.202808 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:36:58.202775 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb53936824fca8eaa6ecd6f840d476533c01fdccedcb54acdcbb3063f83fde0\": container with ID starting with 2fb53936824fca8eaa6ecd6f840d476533c01fdccedcb54acdcbb3063f83fde0 not found: ID does not exist" containerID="2fb53936824fca8eaa6ecd6f840d476533c01fdccedcb54acdcbb3063f83fde0" Nov 03 17:36:58.202907 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:58.202818 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb53936824fca8eaa6ecd6f840d476533c01fdccedcb54acdcbb3063f83fde0"} err="failed to get container status \"2fb53936824fca8eaa6ecd6f840d476533c01fdccedcb54acdcbb3063f83fde0\": rpc error: code = NotFound desc = could not find container \"2fb53936824fca8eaa6ecd6f840d476533c01fdccedcb54acdcbb3063f83fde0\": container with ID starting with 2fb53936824fca8eaa6ecd6f840d476533c01fdccedcb54acdcbb3063f83fde0 not found: ID does not exist" Nov 03 17:36:58.248611 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:58.248565 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r"] Nov 03 17:36:58.260467 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:58.260428 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-460104/distinct-w4-460104-kserve-84bd9f98d4-rcl5r"] Nov 03 17:36:58.566472 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:58.566441 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" path="/var/lib/kubelet/pods/f40eb248-b0bd-43ab-821c-b21ad6f8a364/volumes" Nov 03 17:36:59.936848 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:59.936807 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx"] Nov 03 17:36:59.970515 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:36:59.970469 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp"] Nov 03 17:37:00.131123 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:37:00.131085 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:37:00.838387 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:00.838352 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv"] Nov 03 17:37:00.838667 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:00.838633 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" podUID="4028bf00-f9f7-44ce-9de1-4ffe922e640b" containerName="istio-proxy" containerID="cri-o://75737fefb5726f13651957ace2a3029d8aacd68ea97d0951445dcf4d15888e63" gracePeriod=30 Nov 03 17:37:01.562186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:01.562145 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" podUID="43b5b746-c30a-428b-95ef-7f94cc3f6fd7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:37:02.856071 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:02.856034 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg"] Nov 03 17:37:02.856580 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:02.856320 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" podUID="6739e09f-4597-4915-8400-ecc28a6e122c" containerName="maas-api" containerID="cri-o://3fc5418a8a13253603f5be22716054660f6ead2d05f0c54da321cf32c29ae457" gracePeriod=30 Nov 03 17:37:03.170381 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.170353 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" Nov 03 17:37:03.253092 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.253044 4947 generic.go:356] "Generic (PLEG): container finished" podID="6739e09f-4597-4915-8400-ecc28a6e122c" containerID="3fc5418a8a13253603f5be22716054660f6ead2d05f0c54da321cf32c29ae457" exitCode=0 Nov 03 17:37:03.253302 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.253133 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" event={"ID":"6739e09f-4597-4915-8400-ecc28a6e122c","Type":"ContainerDied","Data":"3fc5418a8a13253603f5be22716054660f6ead2d05f0c54da321cf32c29ae457"} Nov 03 17:37:03.253302 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.253186 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" event={"ID":"6739e09f-4597-4915-8400-ecc28a6e122c","Type":"ContainerDied","Data":"801d361c20005fd3ac5b9c46bfcacbf7c7d99ab6d78ba81bd69a36a7fa95ac9f"} Nov 03 17:37:03.253302 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.253218 4947 scope.go:122] "RemoveContainer" containerID="3fc5418a8a13253603f5be22716054660f6ead2d05f0c54da321cf32c29ae457" Nov 03 17:37:03.253551 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.253397 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg" Nov 03 17:37:03.273015 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.272991 4947 scope.go:122] "RemoveContainer" containerID="3fc5418a8a13253603f5be22716054660f6ead2d05f0c54da321cf32c29ae457" Nov 03 17:37:03.274207 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:37:03.274185 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc5418a8a13253603f5be22716054660f6ead2d05f0c54da321cf32c29ae457\": container with ID starting with 3fc5418a8a13253603f5be22716054660f6ead2d05f0c54da321cf32c29ae457 not found: ID does not exist" containerID="3fc5418a8a13253603f5be22716054660f6ead2d05f0c54da321cf32c29ae457" Nov 03 17:37:03.274357 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.274339 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc5418a8a13253603f5be22716054660f6ead2d05f0c54da321cf32c29ae457"} err="failed to get container status \"3fc5418a8a13253603f5be22716054660f6ead2d05f0c54da321cf32c29ae457\": rpc error: code = NotFound desc = could not find container \"3fc5418a8a13253603f5be22716054660f6ead2d05f0c54da321cf32c29ae457\": container with ID starting with 3fc5418a8a13253603f5be22716054660f6ead2d05f0c54da321cf32c29ae457 not found: ID does not exist" Nov 03 17:37:03.363566 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.363530 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6739e09f-4597-4915-8400-ecc28a6e122c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6739e09f-4597-4915-8400-ecc28a6e122c-maas-api-tls\") pod \"6739e09f-4597-4915-8400-ecc28a6e122c\" (UID: \"6739e09f-4597-4915-8400-ecc28a6e122c\") " Nov 03 17:37:03.363807 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.363588 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6739e09f-4597-4915-8400-ecc28a6e122c-kube-api-access-fb2vr\" (UniqueName: \"kubernetes.io/projected/6739e09f-4597-4915-8400-ecc28a6e122c-kube-api-access-fb2vr\") pod \"6739e09f-4597-4915-8400-ecc28a6e122c\" (UID: \"6739e09f-4597-4915-8400-ecc28a6e122c\") " Nov 03 17:37:03.363807 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.363740 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6739e09f-4597-4915-8400-ecc28a6e122c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6739e09f-4597-4915-8400-ecc28a6e122c-openshift-service-ca\") pod \"6739e09f-4597-4915-8400-ecc28a6e122c\" (UID: \"6739e09f-4597-4915-8400-ecc28a6e122c\") " Nov 03 17:37:03.363939 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.363917 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6739e09f-4597-4915-8400-ecc28a6e122c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6739e09f-4597-4915-8400-ecc28a6e122c-metrics-certs\") pod \"6739e09f-4597-4915-8400-ecc28a6e122c\" (UID: \"6739e09f-4597-4915-8400-ecc28a6e122c\") " Nov 03 17:37:03.364133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.364110 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6739e09f-4597-4915-8400-ecc28a6e122c-openshift-service-ca" pod "6739e09f-4597-4915-8400-ecc28a6e122c" (UID: "6739e09f-4597-4915-8400-ecc28a6e122c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:03.364466 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.364394 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6739e09f-4597-4915-8400-ecc28a6e122c-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:03.366186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.366161 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6739e09f-4597-4915-8400-ecc28a6e122c-kube-api-access-fb2vr" pod "6739e09f-4597-4915-8400-ecc28a6e122c" (UID: "6739e09f-4597-4915-8400-ecc28a6e122c"). InnerVolumeSpecName "kube-api-access-fb2vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:03.366186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.366174 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6739e09f-4597-4915-8400-ecc28a6e122c-maas-api-tls" pod "6739e09f-4597-4915-8400-ecc28a6e122c" (UID: "6739e09f-4597-4915-8400-ecc28a6e122c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:03.366318 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.366237 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6739e09f-4597-4915-8400-ecc28a6e122c-metrics-certs" pod "6739e09f-4597-4915-8400-ecc28a6e122c" (UID: "6739e09f-4597-4915-8400-ecc28a6e122c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:03.465825 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.465741 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb2vr\" (UniqueName: \"kubernetes.io/projected/6739e09f-4597-4915-8400-ecc28a6e122c-kube-api-access-fb2vr\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:03.465825 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.465771 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6739e09f-4597-4915-8400-ecc28a6e122c-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:03.465825 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.465780 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6739e09f-4597-4915-8400-ecc28a6e122c-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:03.604110 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.604074 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg"] Nov 03 17:37:03.617661 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:03.617628 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-a9426b-6b54b8f5f-nr6bg"] Nov 03 17:37:04.564263 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:04.564225 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6739e09f-4597-4915-8400-ecc28a6e122c" path="/var/lib/kubelet/pods/6739e09f-4597-4915-8400-ecc28a6e122c/volumes" Nov 03 17:37:04.940448 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:04.940157 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" podUID="da8b428a-da45-4ad3-ab48-4f2debd415e8" containerName="payload-processing" containerID="cri-o://a031b9a5c71f04507edf6b5150302291b6ed85ebdf952f0ac62ac35f90653e9d" gracePeriod=25 Nov 03 17:37:04.941294 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:04.941268 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w"] Nov 03 17:37:04.942009 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:04.941505 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" podUID="a6a18780-ccc0-4d02-a76b-df4593a94e29" containerName="maas-api" containerID="cri-o://8a50d5fb09d1f79d7d86560a772b19513fc3483d684169c4f43ffa600d44fd74" gracePeriod=30 Nov 03 17:37:04.971307 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:04.971260 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" podUID="cfc2a861-7b75-4723-a8b0-269ab7b28ede" containerName="payload-pre-processing" containerID="cri-o://fa5e85e05517f498468085fcf416733dc130af1962e9a5b183841d0022f28706" gracePeriod=25 Nov 03 17:37:05.267711 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:05.267540 4947 generic.go:356] "Generic (PLEG): container finished" podID="a6a18780-ccc0-4d02-a76b-df4593a94e29" containerID="8a50d5fb09d1f79d7d86560a772b19513fc3483d684169c4f43ffa600d44fd74" exitCode=0 Nov 03 17:37:05.267711 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:05.267605 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" event={"ID":"a6a18780-ccc0-4d02-a76b-df4593a94e29","Type":"ContainerDied","Data":"8a50d5fb09d1f79d7d86560a772b19513fc3483d684169c4f43ffa600d44fd74"} Nov 03 17:37:05.267711 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:05.267670 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" event={"ID":"a6a18780-ccc0-4d02-a76b-df4593a94e29","Type":"ContainerDied","Data":"5dde0ab13e1c25726cb939bef3a86ab8073f651bacfe7dfb04d6fea5876a223f"} Nov 03 17:37:05.267711 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:05.267709 4947 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dde0ab13e1c25726cb939bef3a86ab8073f651bacfe7dfb04d6fea5876a223f" Nov 03 17:37:05.272371 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:05.272344 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" Nov 03 17:37:05.400628 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:05.400591 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6a18780-ccc0-4d02-a76b-df4593a94e29-kube-api-access-n4c4p\" (UniqueName: \"kubernetes.io/projected/a6a18780-ccc0-4d02-a76b-df4593a94e29-kube-api-access-n4c4p\") pod \"a6a18780-ccc0-4d02-a76b-df4593a94e29\" (UID: \"a6a18780-ccc0-4d02-a76b-df4593a94e29\") " Nov 03 17:37:05.400874 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:05.400638 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6a18780-ccc0-4d02-a76b-df4593a94e29-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a18780-ccc0-4d02-a76b-df4593a94e29-maas-api-tls\") pod \"a6a18780-ccc0-4d02-a76b-df4593a94e29\" (UID: \"a6a18780-ccc0-4d02-a76b-df4593a94e29\") " Nov 03 17:37:05.400874 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:05.400811 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6a18780-ccc0-4d02-a76b-df4593a94e29-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6a18780-ccc0-4d02-a76b-df4593a94e29-metrics-certs\") pod \"a6a18780-ccc0-4d02-a76b-df4593a94e29\" (UID: \"a6a18780-ccc0-4d02-a76b-df4593a94e29\") " Nov 03 17:37:05.401019 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:05.400959 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6a18780-ccc0-4d02-a76b-df4593a94e29-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a18780-ccc0-4d02-a76b-df4593a94e29-openshift-service-ca\") pod \"a6a18780-ccc0-4d02-a76b-df4593a94e29\" (UID: \"a6a18780-ccc0-4d02-a76b-df4593a94e29\") " Nov 03 17:37:05.401356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:05.401329 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a18780-ccc0-4d02-a76b-df4593a94e29-openshift-service-ca" pod "a6a18780-ccc0-4d02-a76b-df4593a94e29" (UID: "a6a18780-ccc0-4d02-a76b-df4593a94e29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:05.401637 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:05.401427 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a18780-ccc0-4d02-a76b-df4593a94e29-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:05.403110 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:05.403082 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6a18780-ccc0-4d02-a76b-df4593a94e29-metrics-certs" pod "a6a18780-ccc0-4d02-a76b-df4593a94e29" (UID: "a6a18780-ccc0-4d02-a76b-df4593a94e29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:05.403449 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:05.403426 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6a18780-ccc0-4d02-a76b-df4593a94e29-maas-api-tls" pod "a6a18780-ccc0-4d02-a76b-df4593a94e29" (UID: "a6a18780-ccc0-4d02-a76b-df4593a94e29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:05.403539 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:05.403505 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6a18780-ccc0-4d02-a76b-df4593a94e29-kube-api-access-n4c4p" pod "a6a18780-ccc0-4d02-a76b-df4593a94e29" (UID: "a6a18780-ccc0-4d02-a76b-df4593a94e29"). InnerVolumeSpecName "kube-api-access-n4c4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:05.502324 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:05.502239 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4c4p\" (UniqueName: \"kubernetes.io/projected/a6a18780-ccc0-4d02-a76b-df4593a94e29-kube-api-access-n4c4p\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:05.502324 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:05.502272 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6a18780-ccc0-4d02-a76b-df4593a94e29-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:05.502324 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:05.502281 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6a18780-ccc0-4d02-a76b-df4593a94e29-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:06.114780 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.114686 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 17:37:06.214809 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.214771 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istiod-ca-cert\") pod \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " Nov 03 17:37:06.214973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.214835 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-podinfo\") pod \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " Nov 03 17:37:06.214973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.214933 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-credential-socket\") pod \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " Nov 03 17:37:06.215092 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.215019 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-workload-socket\") pod \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " Nov 03 17:37:06.215092 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.215081 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-data\") pod \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " Nov 03 17:37:06.215196 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.215120 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-token\" (UniqueName: \"kubernetes.io/projected/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-token\") pod \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " Nov 03 17:37:06.215253 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.215187 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istiod-ca-cert" pod "4028bf00-f9f7-44ce-9de1-4ffe922e640b" (UID: "4028bf00-f9f7-44ce-9de1-4ffe922e640b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:06.215253 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.215208 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4028bf00-f9f7-44ce-9de1-4ffe922e640b-kube-api-access-jt2qv\" (UniqueName: \"kubernetes.io/projected/4028bf00-f9f7-44ce-9de1-4ffe922e640b-kube-api-access-jt2qv\") pod \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " Nov 03 17:37:06.215372 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.215270 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-envoy\") pod \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " Nov 03 17:37:06.215372 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.215277 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-workload-socket" pod "4028bf00-f9f7-44ce-9de1-4ffe922e640b" (UID: "4028bf00-f9f7-44ce-9de1-4ffe922e640b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:06.215372 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.215343 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-workload-certs\") pod \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\" (UID: \"4028bf00-f9f7-44ce-9de1-4ffe922e640b\") " Nov 03 17:37:06.215531 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.215363 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-data" pod "4028bf00-f9f7-44ce-9de1-4ffe922e640b" (UID: "4028bf00-f9f7-44ce-9de1-4ffe922e640b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:06.215718 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.215670 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-workload-certs" pod "4028bf00-f9f7-44ce-9de1-4ffe922e640b" (UID: "4028bf00-f9f7-44ce-9de1-4ffe922e640b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:06.215823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.215802 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-credential-socket" pod "4028bf00-f9f7-44ce-9de1-4ffe922e640b" (UID: "4028bf00-f9f7-44ce-9de1-4ffe922e640b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:06.215871 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.215858 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:06.215925 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.215880 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:06.215925 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.215895 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:06.216003 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.215908 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:06.217686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.217659 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4028bf00-f9f7-44ce-9de1-4ffe922e640b-kube-api-access-jt2qv" pod "4028bf00-f9f7-44ce-9de1-4ffe922e640b" (UID: "4028bf00-f9f7-44ce-9de1-4ffe922e640b"). InnerVolumeSpecName "kube-api-access-jt2qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:06.217686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.217671 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-token" pod "4028bf00-f9f7-44ce-9de1-4ffe922e640b" (UID: "4028bf00-f9f7-44ce-9de1-4ffe922e640b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:06.217935 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.217917 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-podinfo" pod "4028bf00-f9f7-44ce-9de1-4ffe922e640b" (UID: "4028bf00-f9f7-44ce-9de1-4ffe922e640b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:37:06.218080 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.218055 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-envoy" pod "4028bf00-f9f7-44ce-9de1-4ffe922e640b" (UID: "4028bf00-f9f7-44ce-9de1-4ffe922e640b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:06.276101 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.276066 4947 generic.go:356] "Generic (PLEG): container finished" podID="4028bf00-f9f7-44ce-9de1-4ffe922e640b" containerID="75737fefb5726f13651957ace2a3029d8aacd68ea97d0951445dcf4d15888e63" exitCode=0 Nov 03 17:37:06.276297 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.276130 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" Nov 03 17:37:06.276297 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.276149 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" event={"ID":"4028bf00-f9f7-44ce-9de1-4ffe922e640b","Type":"ContainerDied","Data":"75737fefb5726f13651957ace2a3029d8aacd68ea97d0951445dcf4d15888e63"} Nov 03 17:37:06.276297 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.276199 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv" event={"ID":"4028bf00-f9f7-44ce-9de1-4ffe922e640b","Type":"ContainerDied","Data":"7bd4a1cb085a67da20bc7fd03b4a8b344f571c0a9c83ffdb5d39cd7f73e6927c"} Nov 03 17:37:06.276297 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.276222 4947 scope.go:122] "RemoveContainer" containerID="75737fefb5726f13651957ace2a3029d8aacd68ea97d0951445dcf4d15888e63" Nov 03 17:37:06.276606 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.276382 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w" Nov 03 17:37:06.286337 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.286319 4947 scope.go:122] "RemoveContainer" containerID="75737fefb5726f13651957ace2a3029d8aacd68ea97d0951445dcf4d15888e63" Nov 03 17:37:06.286640 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:37:06.286615 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75737fefb5726f13651957ace2a3029d8aacd68ea97d0951445dcf4d15888e63\": container with ID starting with 75737fefb5726f13651957ace2a3029d8aacd68ea97d0951445dcf4d15888e63 not found: ID does not exist" containerID="75737fefb5726f13651957ace2a3029d8aacd68ea97d0951445dcf4d15888e63" Nov 03 17:37:06.286736 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.286649 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75737fefb5726f13651957ace2a3029d8aacd68ea97d0951445dcf4d15888e63"} err="failed to get container status \"75737fefb5726f13651957ace2a3029d8aacd68ea97d0951445dcf4d15888e63\": rpc error: code = NotFound desc = could not find container \"75737fefb5726f13651957ace2a3029d8aacd68ea97d0951445dcf4d15888e63\": container with ID starting with 75737fefb5726f13651957ace2a3029d8aacd68ea97d0951445dcf4d15888e63 not found: ID does not exist" Nov 03 17:37:06.317441 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.317413 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt2qv\" (UniqueName: \"kubernetes.io/projected/4028bf00-f9f7-44ce-9de1-4ffe922e640b-kube-api-access-jt2qv\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:06.317441 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.317441 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:06.317609 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.317453 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:06.317609 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.317462 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4028bf00-f9f7-44ce-9de1-4ffe922e640b-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:06.317609 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.317473 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4028bf00-f9f7-44ce-9de1-4ffe922e640b-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:06.337770 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.337745 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w"] Nov 03 17:37:06.342814 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.342792 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-460104-6f554bc9cd-ht89w"] Nov 03 17:37:06.379314 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.379228 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv"] Nov 03 17:37:06.385950 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.385922 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-71d7b4-openshift-default-555db9fddc-6b8cv"] Nov 03 17:37:06.570903 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.570865 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4028bf00-f9f7-44ce-9de1-4ffe922e640b" path="/var/lib/kubelet/pods/4028bf00-f9f7-44ce-9de1-4ffe922e640b/volumes" Nov 03 17:37:06.571315 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:06.571298 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6a18780-ccc0-4d02-a76b-df4593a94e29" path="/var/lib/kubelet/pods/a6a18780-ccc0-4d02-a76b-df4593a94e29/volumes" Nov 03 17:37:07.184447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:07.184385 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" Nov 03 17:37:07.227678 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:07.227643 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b5b746-c30a-428b-95ef-7f94cc3f6fd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b5b746-c30a-428b-95ef-7f94cc3f6fd7-plugins-config-volume\") pod \"43b5b746-c30a-428b-95ef-7f94cc3f6fd7\" (UID: \"43b5b746-c30a-428b-95ef-7f94cc3f6fd7\") " Nov 03 17:37:07.227863 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:07.227764 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b5b746-c30a-428b-95ef-7f94cc3f6fd7-kube-api-access-5ddp6\" (UniqueName: \"kubernetes.io/projected/43b5b746-c30a-428b-95ef-7f94cc3f6fd7-kube-api-access-5ddp6\") pod \"43b5b746-c30a-428b-95ef-7f94cc3f6fd7\" (UID: \"43b5b746-c30a-428b-95ef-7f94cc3f6fd7\") " Nov 03 17:37:07.228052 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:07.228032 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b5b746-c30a-428b-95ef-7f94cc3f6fd7-plugins-config-volume" pod "43b5b746-c30a-428b-95ef-7f94cc3f6fd7" (UID: "43b5b746-c30a-428b-95ef-7f94cc3f6fd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:07.230095 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:07.230066 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b5b746-c30a-428b-95ef-7f94cc3f6fd7-kube-api-access-5ddp6" pod "43b5b746-c30a-428b-95ef-7f94cc3f6fd7" (UID: "43b5b746-c30a-428b-95ef-7f94cc3f6fd7"). InnerVolumeSpecName "kube-api-access-5ddp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:07.283507 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:07.283474 4947 generic.go:356] "Generic (PLEG): container finished" podID="43b5b746-c30a-428b-95ef-7f94cc3f6fd7" containerID="771f33136531bc0165308faaae1d3d5be0ae19b76651ae87a894a190e35d0f1f" exitCode=0 Nov 03 17:37:07.283686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:07.283518 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" Nov 03 17:37:07.283686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:07.283551 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" event={"ID":"43b5b746-c30a-428b-95ef-7f94cc3f6fd7","Type":"ContainerDied","Data":"771f33136531bc0165308faaae1d3d5be0ae19b76651ae87a894a190e35d0f1f"} Nov 03 17:37:07.283686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:07.283602 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf" event={"ID":"43b5b746-c30a-428b-95ef-7f94cc3f6fd7","Type":"ContainerDied","Data":"7f462c5292436839251f629e292e085d8bc23c52cb5695c8cabf1dd191e95f79"} Nov 03 17:37:07.283686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:07.283623 4947 scope.go:122] "RemoveContainer" containerID="771f33136531bc0165308faaae1d3d5be0ae19b76651ae87a894a190e35d0f1f" Nov 03 17:37:07.293473 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:07.293443 4947 scope.go:122] "RemoveContainer" containerID="771f33136531bc0165308faaae1d3d5be0ae19b76651ae87a894a190e35d0f1f" Nov 03 17:37:07.293788 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:37:07.293769 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"771f33136531bc0165308faaae1d3d5be0ae19b76651ae87a894a190e35d0f1f\": container with ID starting with 771f33136531bc0165308faaae1d3d5be0ae19b76651ae87a894a190e35d0f1f not found: ID does not exist" containerID="771f33136531bc0165308faaae1d3d5be0ae19b76651ae87a894a190e35d0f1f" Nov 03 17:37:07.293874 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:07.293797 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"771f33136531bc0165308faaae1d3d5be0ae19b76651ae87a894a190e35d0f1f"} err="failed to get container status \"771f33136531bc0165308faaae1d3d5be0ae19b76651ae87a894a190e35d0f1f\": rpc error: code = NotFound desc = could not find container \"771f33136531bc0165308faaae1d3d5be0ae19b76651ae87a894a190e35d0f1f\": container with ID starting with 771f33136531bc0165308faaae1d3d5be0ae19b76651ae87a894a190e35d0f1f not found: ID does not exist" Nov 03 17:37:07.329571 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:07.329545 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43b5b746-c30a-428b-95ef-7f94cc3f6fd7-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:07.329571 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:07.329574 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ddp6\" (UniqueName: \"kubernetes.io/projected/43b5b746-c30a-428b-95ef-7f94cc3f6fd7-kube-api-access-5ddp6\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:07.345931 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:07.345901 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf"] Nov 03 17:37:07.355850 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:07.355828 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-790111-7ff4695579-wnqhf"] Nov 03 17:37:08.305408 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:08.305375 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg"] Nov 03 17:37:08.306134 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:08.305622 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" podUID="783635c1-dfe9-4471-851a-e0b3c8ee21c5" containerName="istio-proxy" containerID="cri-o://b22f11a6e8d44d35203f747de62c8e1cfbbbb28c18859f321c109003f94278d9" gracePeriod=30 Nov 03 17:37:08.564925 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:08.564847 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b5b746-c30a-428b-95ef-7f94cc3f6fd7" path="/var/lib/kubelet/pods/43b5b746-c30a-428b-95ef-7f94cc3f6fd7/volumes" Nov 03 17:37:09.708722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:09.704539 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:37:13.567005 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.566978 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:37:13.602633 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.602568 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-workload-certs\") pod \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " Nov 03 17:37:13.602633 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.602614 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/783635c1-dfe9-4471-851a-e0b3c8ee21c5-kube-api-access-d6v2l\" (UniqueName: \"kubernetes.io/projected/783635c1-dfe9-4471-851a-e0b3c8ee21c5-kube-api-access-d6v2l\") pod \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " Nov 03 17:37:13.602851 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.602662 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-data\") pod \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " Nov 03 17:37:13.602851 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.602682 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-envoy\") pod \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " Nov 03 17:37:13.602851 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.602792 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-workload-socket\") pod \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " Nov 03 17:37:13.602851 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.602821 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-podinfo\") pod \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " Nov 03 17:37:13.603032 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.602845 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-workload-certs" pod "783635c1-dfe9-4471-851a-e0b3c8ee21c5" (UID: "783635c1-dfe9-4471-851a-e0b3c8ee21c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:13.603032 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.602888 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-token\" (UniqueName: \"kubernetes.io/projected/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-token\") pod \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " Nov 03 17:37:13.603032 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.602946 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-credential-socket\") pod \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " Nov 03 17:37:13.603032 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.602955 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-data" pod "783635c1-dfe9-4471-851a-e0b3c8ee21c5" (UID: "783635c1-dfe9-4471-851a-e0b3c8ee21c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:13.603032 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.602977 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istiod-ca-cert\") pod \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\" (UID: \"783635c1-dfe9-4471-851a-e0b3c8ee21c5\") " Nov 03 17:37:13.603322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.603076 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-workload-socket" pod "783635c1-dfe9-4471-851a-e0b3c8ee21c5" (UID: "783635c1-dfe9-4471-851a-e0b3c8ee21c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:13.603322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.603178 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-credential-socket" pod "783635c1-dfe9-4471-851a-e0b3c8ee21c5" (UID: "783635c1-dfe9-4471-851a-e0b3c8ee21c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:13.603548 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.603463 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:13.603548 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.603488 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:13.603548 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.603502 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:13.603548 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.603513 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:13.603899 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.603595 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istiod-ca-cert" pod "783635c1-dfe9-4471-851a-e0b3c8ee21c5" (UID: "783635c1-dfe9-4471-851a-e0b3c8ee21c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:13.605124 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.605095 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-envoy" pod "783635c1-dfe9-4471-851a-e0b3c8ee21c5" (UID: "783635c1-dfe9-4471-851a-e0b3c8ee21c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:13.605224 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.605153 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783635c1-dfe9-4471-851a-e0b3c8ee21c5-kube-api-access-d6v2l" pod "783635c1-dfe9-4471-851a-e0b3c8ee21c5" (UID: "783635c1-dfe9-4471-851a-e0b3c8ee21c5"). InnerVolumeSpecName "kube-api-access-d6v2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:13.605630 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.605607 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-podinfo" pod "783635c1-dfe9-4471-851a-e0b3c8ee21c5" (UID: "783635c1-dfe9-4471-851a-e0b3c8ee21c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:37:13.605769 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.605742 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-token" pod "783635c1-dfe9-4471-851a-e0b3c8ee21c5" (UID: "783635c1-dfe9-4471-851a-e0b3c8ee21c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:13.704581 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.704543 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:13.704581 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.704575 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:13.704581 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.704583 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:13.704858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.704591 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6v2l\" (UniqueName: \"kubernetes.io/projected/783635c1-dfe9-4471-851a-e0b3c8ee21c5-kube-api-access-d6v2l\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:13.704858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:13.704601 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/783635c1-dfe9-4471-851a-e0b3c8ee21c5-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:14.156541 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:14.156502 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" podUID="da8b428a-da45-4ad3-ab48-4f2debd415e8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:37:14.156784 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:14.156505 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" podUID="cfc2a861-7b75-4723-a8b0-269ab7b28ede" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:37:14.338167 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:14.338134 4947 generic.go:356] "Generic (PLEG): container finished" podID="783635c1-dfe9-4471-851a-e0b3c8ee21c5" containerID="b22f11a6e8d44d35203f747de62c8e1cfbbbb28c18859f321c109003f94278d9" exitCode=0 Nov 03 17:37:14.338393 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:14.338190 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" Nov 03 17:37:14.338393 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:14.338218 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" event={"ID":"783635c1-dfe9-4471-851a-e0b3c8ee21c5","Type":"ContainerDied","Data":"b22f11a6e8d44d35203f747de62c8e1cfbbbb28c18859f321c109003f94278d9"} Nov 03 17:37:14.338393 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:14.338276 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg" event={"ID":"783635c1-dfe9-4471-851a-e0b3c8ee21c5","Type":"ContainerDied","Data":"53b6e75c653ad26c7374d60b3b0fcbe81e054d9659117c28f351b6adc439f391"} Nov 03 17:37:14.338393 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:14.338303 4947 scope.go:122] "RemoveContainer" containerID="b22f11a6e8d44d35203f747de62c8e1cfbbbb28c18859f321c109003f94278d9" Nov 03 17:37:14.348664 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:14.348647 4947 scope.go:122] "RemoveContainer" containerID="b22f11a6e8d44d35203f747de62c8e1cfbbbb28c18859f321c109003f94278d9" Nov 03 17:37:14.348944 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:37:14.348926 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b22f11a6e8d44d35203f747de62c8e1cfbbbb28c18859f321c109003f94278d9\": container with ID starting with b22f11a6e8d44d35203f747de62c8e1cfbbbb28c18859f321c109003f94278d9 not found: ID does not exist" containerID="b22f11a6e8d44d35203f747de62c8e1cfbbbb28c18859f321c109003f94278d9" Nov 03 17:37:14.349025 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:14.348957 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b22f11a6e8d44d35203f747de62c8e1cfbbbb28c18859f321c109003f94278d9"} err="failed to get container status \"b22f11a6e8d44d35203f747de62c8e1cfbbbb28c18859f321c109003f94278d9\": rpc error: code = NotFound desc = could not find container \"b22f11a6e8d44d35203f747de62c8e1cfbbbb28c18859f321c109003f94278d9\": container with ID starting with b22f11a6e8d44d35203f747de62c8e1cfbbbb28c18859f321c109003f94278d9 not found: ID does not exist" Nov 03 17:37:14.413913 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:14.413822 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg"] Nov 03 17:37:14.421185 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:14.421158 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-a9426b-openshift-default-7c567466b5-rd8bg"] Nov 03 17:37:14.564493 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:14.564460 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="783635c1-dfe9-4471-851a-e0b3c8ee21c5" path="/var/lib/kubelet/pods/783635c1-dfe9-4471-851a-e0b3c8ee21c5/volumes" Nov 03 17:37:19.736998 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:19.736961 4947 scope.go:122] "RemoveContainer" containerID="8a50d5fb09d1f79d7d86560a772b19513fc3483d684169c4f43ffa600d44fd74" Nov 03 17:37:20.100787 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:20.100755 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:37:20.269242 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:20.269201 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf"] Nov 03 17:37:20.269490 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:20.269463 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" podUID="64013461-1f82-44ce-9c7e-6dfc9d4b9159" containerName="istio-proxy" containerID="cri-o://a9f95c224bfabdde8020502c316fffcd2d74e7f536f6f0e34f44dfafa2e90a80" gracePeriod=30 Nov 03 17:37:22.818940 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:22.818901 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84"] Nov 03 17:37:22.819490 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:22.819171 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" podUID="f2bd4293-1c17-455d-9237-38edc12c9890" containerName="istio-proxy" containerID="cri-o://9088ac672a34d90115e6063e35401fc27e3e6e4e5ef995344bedfe610c88a293" gracePeriod=30 Nov 03 17:37:24.156276 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:24.156240 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" podUID="cfc2a861-7b75-4723-a8b0-269ab7b28ede" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:37:24.156673 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:24.156279 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" podUID="da8b428a-da45-4ad3-ab48-4f2debd415e8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:37:24.689810 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:24.689774 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4"] Nov 03 17:37:24.690033 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:24.690004 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" podUID="3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" containerName="istio-proxy" containerID="cri-o://7c1360f18cf9a93c9bf64d8ce4970168d9f9c88b99c4bafb4a0ed28e0b65b818" gracePeriod=30 Nov 03 17:37:25.256345 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.256308 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" Nov 03 17:37:25.345711 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.345665 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" Nov 03 17:37:25.378724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.378677 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8b428a-da45-4ad3-ab48-4f2debd415e8-kube-api-access-8p9n7\" (UniqueName: \"kubernetes.io/projected/da8b428a-da45-4ad3-ab48-4f2debd415e8-kube-api-access-8p9n7\") pod \"da8b428a-da45-4ad3-ab48-4f2debd415e8\" (UID: \"da8b428a-da45-4ad3-ab48-4f2debd415e8\") " Nov 03 17:37:25.378897 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.378862 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da8b428a-da45-4ad3-ab48-4f2debd415e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8b428a-da45-4ad3-ab48-4f2debd415e8-plugins-config-volume\") pod \"da8b428a-da45-4ad3-ab48-4f2debd415e8\" (UID: \"da8b428a-da45-4ad3-ab48-4f2debd415e8\") " Nov 03 17:37:25.379259 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.379233 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da8b428a-da45-4ad3-ab48-4f2debd415e8-plugins-config-volume" pod "da8b428a-da45-4ad3-ab48-4f2debd415e8" (UID: "da8b428a-da45-4ad3-ab48-4f2debd415e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:25.381015 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.380991 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8b428a-da45-4ad3-ab48-4f2debd415e8-kube-api-access-8p9n7" pod "da8b428a-da45-4ad3-ab48-4f2debd415e8" (UID: "da8b428a-da45-4ad3-ab48-4f2debd415e8"). InnerVolumeSpecName "kube-api-access-8p9n7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:25.422969 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.422931 4947 generic.go:356] "Generic (PLEG): container finished" podID="da8b428a-da45-4ad3-ab48-4f2debd415e8" containerID="a031b9a5c71f04507edf6b5150302291b6ed85ebdf952f0ac62ac35f90653e9d" exitCode=0 Nov 03 17:37:25.423186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.422983 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" Nov 03 17:37:25.423186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.423035 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" event={"ID":"da8b428a-da45-4ad3-ab48-4f2debd415e8","Type":"ContainerDied","Data":"a031b9a5c71f04507edf6b5150302291b6ed85ebdf952f0ac62ac35f90653e9d"} Nov 03 17:37:25.423186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.423101 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx" event={"ID":"da8b428a-da45-4ad3-ab48-4f2debd415e8","Type":"ContainerDied","Data":"1a94b22fc26f5e6a9d2c271a43512f44489d5a098b4ad3f63bfc19e97aba732e"} Nov 03 17:37:25.423186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.423136 4947 scope.go:122] "RemoveContainer" containerID="a031b9a5c71f04507edf6b5150302291b6ed85ebdf952f0ac62ac35f90653e9d" Nov 03 17:37:25.425883 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.425850 4947 generic.go:356] "Generic (PLEG): container finished" podID="64013461-1f82-44ce-9c7e-6dfc9d4b9159" containerID="a9f95c224bfabdde8020502c316fffcd2d74e7f536f6f0e34f44dfafa2e90a80" exitCode=0 Nov 03 17:37:25.426003 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.425957 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" event={"ID":"64013461-1f82-44ce-9c7e-6dfc9d4b9159","Type":"ContainerDied","Data":"a9f95c224bfabdde8020502c316fffcd2d74e7f536f6f0e34f44dfafa2e90a80"} Nov 03 17:37:25.427808 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.427742 4947 generic.go:356] "Generic (PLEG): container finished" podID="cfc2a861-7b75-4723-a8b0-269ab7b28ede" containerID="fa5e85e05517f498468085fcf416733dc130af1962e9a5b183841d0022f28706" exitCode=0 Nov 03 17:37:25.427808 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.427778 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" event={"ID":"cfc2a861-7b75-4723-a8b0-269ab7b28ede","Type":"ContainerDied","Data":"fa5e85e05517f498468085fcf416733dc130af1962e9a5b183841d0022f28706"} Nov 03 17:37:25.428003 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.427837 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" event={"ID":"cfc2a861-7b75-4723-a8b0-269ab7b28ede","Type":"ContainerDied","Data":"dca1e00118040293fa3d1d961a9407fb0c576769b452c9a10b3fa131382ef808"} Nov 03 17:37:25.428003 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.427844 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp" Nov 03 17:37:25.438092 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.438067 4947 scope.go:122] "RemoveContainer" containerID="a031b9a5c71f04507edf6b5150302291b6ed85ebdf952f0ac62ac35f90653e9d" Nov 03 17:37:25.438412 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:37:25.438382 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a031b9a5c71f04507edf6b5150302291b6ed85ebdf952f0ac62ac35f90653e9d\": container with ID starting with a031b9a5c71f04507edf6b5150302291b6ed85ebdf952f0ac62ac35f90653e9d not found: ID does not exist" containerID="a031b9a5c71f04507edf6b5150302291b6ed85ebdf952f0ac62ac35f90653e9d" Nov 03 17:37:25.438499 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.438423 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a031b9a5c71f04507edf6b5150302291b6ed85ebdf952f0ac62ac35f90653e9d"} err="failed to get container status \"a031b9a5c71f04507edf6b5150302291b6ed85ebdf952f0ac62ac35f90653e9d\": rpc error: code = NotFound desc = could not find container \"a031b9a5c71f04507edf6b5150302291b6ed85ebdf952f0ac62ac35f90653e9d\": container with ID starting with a031b9a5c71f04507edf6b5150302291b6ed85ebdf952f0ac62ac35f90653e9d not found: ID does not exist" Nov 03 17:37:25.438499 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.438448 4947 scope.go:122] "RemoveContainer" containerID="fa5e85e05517f498468085fcf416733dc130af1962e9a5b183841d0022f28706" Nov 03 17:37:25.448665 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.448644 4947 scope.go:122] "RemoveContainer" containerID="fa5e85e05517f498468085fcf416733dc130af1962e9a5b183841d0022f28706" Nov 03 17:37:25.449043 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:37:25.449022 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa5e85e05517f498468085fcf416733dc130af1962e9a5b183841d0022f28706\": container with ID starting with fa5e85e05517f498468085fcf416733dc130af1962e9a5b183841d0022f28706 not found: ID does not exist" containerID="fa5e85e05517f498468085fcf416733dc130af1962e9a5b183841d0022f28706" Nov 03 17:37:25.449112 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.449053 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa5e85e05517f498468085fcf416733dc130af1962e9a5b183841d0022f28706"} err="failed to get container status \"fa5e85e05517f498468085fcf416733dc130af1962e9a5b183841d0022f28706\": rpc error: code = NotFound desc = could not find container \"fa5e85e05517f498468085fcf416733dc130af1962e9a5b183841d0022f28706\": container with ID starting with fa5e85e05517f498468085fcf416733dc130af1962e9a5b183841d0022f28706 not found: ID does not exist" Nov 03 17:37:25.480266 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.480188 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx"] Nov 03 17:37:25.480266 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.480238 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc2a861-7b75-4723-a8b0-269ab7b28ede-kube-api-access-gfwt4\" (UniqueName: \"kubernetes.io/projected/cfc2a861-7b75-4723-a8b0-269ab7b28ede-kube-api-access-gfwt4\") pod \"cfc2a861-7b75-4723-a8b0-269ab7b28ede\" (UID: \"cfc2a861-7b75-4723-a8b0-269ab7b28ede\") " Nov 03 17:37:25.480497 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.480479 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc2a861-7b75-4723-a8b0-269ab7b28ede-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc2a861-7b75-4723-a8b0-269ab7b28ede-plugins-config-volume\") pod \"cfc2a861-7b75-4723-a8b0-269ab7b28ede\" (UID: \"cfc2a861-7b75-4723-a8b0-269ab7b28ede\") " Nov 03 17:37:25.481514 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.481486 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p9n7\" (UniqueName: \"kubernetes.io/projected/da8b428a-da45-4ad3-ab48-4f2debd415e8-kube-api-access-8p9n7\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.481684 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.481516 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da8b428a-da45-4ad3-ab48-4f2debd415e8-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.484052 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.484023 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc2a861-7b75-4723-a8b0-269ab7b28ede-kube-api-access-gfwt4" pod "cfc2a861-7b75-4723-a8b0-269ab7b28ede" (UID: "cfc2a861-7b75-4723-a8b0-269ab7b28ede"). InnerVolumeSpecName "kube-api-access-gfwt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:25.484984 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.484957 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc2a861-7b75-4723-a8b0-269ab7b28ede-plugins-config-volume" pod "cfc2a861-7b75-4723-a8b0-269ab7b28ede" (UID: "cfc2a861-7b75-4723-a8b0-269ab7b28ede"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:25.485497 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.485476 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-a9426b-86c859947-lnfhx"] Nov 03 17:37:25.527885 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.527859 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:37:25.582192 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.582160 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfc2a861-7b75-4723-a8b0-269ab7b28ede-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.582192 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.582188 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfwt4\" (UniqueName: \"kubernetes.io/projected/cfc2a861-7b75-4723-a8b0-269ab7b28ede-kube-api-access-gfwt4\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.682905 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.682874 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istiod-ca-cert\") pod \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " Nov 03 17:37:25.683100 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.682910 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-token\" (UniqueName: \"kubernetes.io/projected/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-token\") pod \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " Nov 03 17:37:25.683100 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.682985 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-data\") pod \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " Nov 03 17:37:25.683100 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.683021 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64013461-1f82-44ce-9c7e-6dfc9d4b9159-kube-api-access-xvnb8\" (UniqueName: \"kubernetes.io/projected/64013461-1f82-44ce-9c7e-6dfc9d4b9159-kube-api-access-xvnb8\") pod \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " Nov 03 17:37:25.683100 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.683052 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-envoy\") pod \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " Nov 03 17:37:25.683100 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.683098 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-workload-certs\") pod \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " Nov 03 17:37:25.683429 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.683119 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-podinfo\") pod \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " Nov 03 17:37:25.683429 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.683181 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-credential-socket\") pod \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " Nov 03 17:37:25.683429 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.683248 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-workload-socket\") pod \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\" (UID: \"64013461-1f82-44ce-9c7e-6dfc9d4b9159\") " Nov 03 17:37:25.683429 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.683353 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-data" pod "64013461-1f82-44ce-9c7e-6dfc9d4b9159" (UID: "64013461-1f82-44ce-9c7e-6dfc9d4b9159"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:25.683429 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.683393 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istiod-ca-cert" pod "64013461-1f82-44ce-9c7e-6dfc9d4b9159" (UID: "64013461-1f82-44ce-9c7e-6dfc9d4b9159"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:25.683722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.683581 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-credential-socket" pod "64013461-1f82-44ce-9c7e-6dfc9d4b9159" (UID: "64013461-1f82-44ce-9c7e-6dfc9d4b9159"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:25.683801 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.683749 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.683801 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.683763 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.683801 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.683755 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-workload-certs" pod "64013461-1f82-44ce-9c7e-6dfc9d4b9159" (UID: "64013461-1f82-44ce-9c7e-6dfc9d4b9159"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:25.683801 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.683772 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.684199 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.683873 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-workload-socket" pod "64013461-1f82-44ce-9c7e-6dfc9d4b9159" (UID: "64013461-1f82-44ce-9c7e-6dfc9d4b9159"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:25.685598 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.685570 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64013461-1f82-44ce-9c7e-6dfc9d4b9159-kube-api-access-xvnb8" pod "64013461-1f82-44ce-9c7e-6dfc9d4b9159" (UID: "64013461-1f82-44ce-9c7e-6dfc9d4b9159"). InnerVolumeSpecName "kube-api-access-xvnb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:25.685598 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.685583 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-token" pod "64013461-1f82-44ce-9c7e-6dfc9d4b9159" (UID: "64013461-1f82-44ce-9c7e-6dfc9d4b9159"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:25.686015 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.685998 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-podinfo" pod "64013461-1f82-44ce-9c7e-6dfc9d4b9159" (UID: "64013461-1f82-44ce-9c7e-6dfc9d4b9159"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:37:25.686066 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.686029 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-envoy" pod "64013461-1f82-44ce-9c7e-6dfc9d4b9159" (UID: "64013461-1f82-44ce-9c7e-6dfc9d4b9159"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:25.776093 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.776055 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp"] Nov 03 17:37:25.779827 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.779801 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-a9426b-7655cd4c79-l4qkp"] Nov 03 17:37:25.785390 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.785365 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvnb8\" (UniqueName: \"kubernetes.io/projected/64013461-1f82-44ce-9c7e-6dfc9d4b9159-kube-api-access-xvnb8\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.785390 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.785390 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.785612 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.785401 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.785612 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.785410 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.785612 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.785418 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64013461-1f82-44ce-9c7e-6dfc9d4b9159-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:25.785612 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:25.785426 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64013461-1f82-44ce-9c7e-6dfc9d4b9159-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:26.435961 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:26.435925 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" event={"ID":"64013461-1f82-44ce-9c7e-6dfc9d4b9159","Type":"ContainerDied","Data":"5527e745e6636f35476472418d8766cd7cc0d7aa36fbd0aac81bb6da8bdd16a4"} Nov 03 17:37:26.436470 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:26.435934 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf" Nov 03 17:37:26.436470 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:26.436011 4947 scope.go:122] "RemoveContainer" containerID="a9f95c224bfabdde8020502c316fffcd2d74e7f536f6f0e34f44dfafa2e90a80" Nov 03 17:37:26.505260 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:26.505226 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf"] Nov 03 17:37:26.518454 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:26.518430 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-z7szf"] Nov 03 17:37:26.564877 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:26.564836 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64013461-1f82-44ce-9c7e-6dfc9d4b9159" path="/var/lib/kubelet/pods/64013461-1f82-44ce-9c7e-6dfc9d4b9159/volumes" Nov 03 17:37:26.565306 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:26.565283 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc2a861-7b75-4723-a8b0-269ab7b28ede" path="/var/lib/kubelet/pods/cfc2a861-7b75-4723-a8b0-269ab7b28ede/volumes" Nov 03 17:37:26.565779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:26.565756 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8b428a-da45-4ad3-ab48-4f2debd415e8" path="/var/lib/kubelet/pods/da8b428a-da45-4ad3-ab48-4f2debd415e8/volumes" Nov 03 17:37:28.086396 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.086341 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 17:37:28.209903 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.209848 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2bd4293-1c17-455d-9237-38edc12c9890-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2bd4293-1c17-455d-9237-38edc12c9890-istio-podinfo\") pod \"f2bd4293-1c17-455d-9237-38edc12c9890\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " Nov 03 17:37:28.209903 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.209912 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-istio-envoy\") pod \"f2bd4293-1c17-455d-9237-38edc12c9890\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " Nov 03 17:37:28.210158 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.209968 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2bd4293-1c17-455d-9237-38edc12c9890-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2bd4293-1c17-455d-9237-38edc12c9890-istiod-ca-cert\") pod \"f2bd4293-1c17-455d-9237-38edc12c9890\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " Nov 03 17:37:28.210158 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.210040 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2bd4293-1c17-455d-9237-38edc12c9890-istio-token\" (UniqueName: \"kubernetes.io/projected/f2bd4293-1c17-455d-9237-38edc12c9890-istio-token\") pod \"f2bd4293-1c17-455d-9237-38edc12c9890\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " Nov 03 17:37:28.210158 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.210067 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2bd4293-1c17-455d-9237-38edc12c9890-kube-api-access-bxw76\" (UniqueName: \"kubernetes.io/projected/f2bd4293-1c17-455d-9237-38edc12c9890-kube-api-access-bxw76\") pod \"f2bd4293-1c17-455d-9237-38edc12c9890\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " Nov 03 17:37:28.210158 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.210093 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-credential-socket\") pod \"f2bd4293-1c17-455d-9237-38edc12c9890\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " Nov 03 17:37:28.210158 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.210119 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-workload-certs\") pod \"f2bd4293-1c17-455d-9237-38edc12c9890\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " Nov 03 17:37:28.210534 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.210161 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-istio-data\") pod \"f2bd4293-1c17-455d-9237-38edc12c9890\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " Nov 03 17:37:28.210534 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.210251 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-workload-socket\") pod \"f2bd4293-1c17-455d-9237-38edc12c9890\" (UID: \"f2bd4293-1c17-455d-9237-38edc12c9890\") " Nov 03 17:37:28.210534 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.210456 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2bd4293-1c17-455d-9237-38edc12c9890-istiod-ca-cert" pod "f2bd4293-1c17-455d-9237-38edc12c9890" (UID: "f2bd4293-1c17-455d-9237-38edc12c9890"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:28.210737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.210646 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2bd4293-1c17-455d-9237-38edc12c9890-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:28.210737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.210691 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-istio-data" pod "f2bd4293-1c17-455d-9237-38edc12c9890" (UID: "f2bd4293-1c17-455d-9237-38edc12c9890"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:28.210951 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.210893 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-workload-socket" pod "f2bd4293-1c17-455d-9237-38edc12c9890" (UID: "f2bd4293-1c17-455d-9237-38edc12c9890"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:28.211047 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.210993 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-credential-socket" pod "f2bd4293-1c17-455d-9237-38edc12c9890" (UID: "f2bd4293-1c17-455d-9237-38edc12c9890"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:28.211047 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.211025 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-workload-certs" pod "f2bd4293-1c17-455d-9237-38edc12c9890" (UID: "f2bd4293-1c17-455d-9237-38edc12c9890"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:28.212832 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.212808 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2bd4293-1c17-455d-9237-38edc12c9890-kube-api-access-bxw76" pod "f2bd4293-1c17-455d-9237-38edc12c9890" (UID: "f2bd4293-1c17-455d-9237-38edc12c9890"). InnerVolumeSpecName "kube-api-access-bxw76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:28.212956 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.212908 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-istio-envoy" pod "f2bd4293-1c17-455d-9237-38edc12c9890" (UID: "f2bd4293-1c17-455d-9237-38edc12c9890"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:28.213020 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.212952 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2bd4293-1c17-455d-9237-38edc12c9890-istio-token" pod "f2bd4293-1c17-455d-9237-38edc12c9890" (UID: "f2bd4293-1c17-455d-9237-38edc12c9890"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:28.213020 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.213010 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2bd4293-1c17-455d-9237-38edc12c9890-istio-podinfo" pod "f2bd4293-1c17-455d-9237-38edc12c9890" (UID: "f2bd4293-1c17-455d-9237-38edc12c9890"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:37:28.311970 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.311934 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxw76\" (UniqueName: \"kubernetes.io/projected/f2bd4293-1c17-455d-9237-38edc12c9890-kube-api-access-bxw76\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:28.311970 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.311973 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:28.312180 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.311987 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:28.312180 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.312001 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:28.312180 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.312016 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:28.312180 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.312029 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2bd4293-1c17-455d-9237-38edc12c9890-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:28.312180 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.312042 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2bd4293-1c17-455d-9237-38edc12c9890-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:28.312180 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.312054 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2bd4293-1c17-455d-9237-38edc12c9890-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:28.449376 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.449283 4947 generic.go:356] "Generic (PLEG): container finished" podID="f2bd4293-1c17-455d-9237-38edc12c9890" containerID="9088ac672a34d90115e6063e35401fc27e3e6e4e5ef995344bedfe610c88a293" exitCode=0 Nov 03 17:37:28.449376 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.449346 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" Nov 03 17:37:28.449748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.449387 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" event={"ID":"f2bd4293-1c17-455d-9237-38edc12c9890","Type":"ContainerDied","Data":"9088ac672a34d90115e6063e35401fc27e3e6e4e5ef995344bedfe610c88a293"} Nov 03 17:37:28.449748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.449455 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84" event={"ID":"f2bd4293-1c17-455d-9237-38edc12c9890","Type":"ContainerDied","Data":"d39a07332468bcbf0fb3565ce4a28061bbf4887b0728a9333bc60a1dd8263468"} Nov 03 17:37:28.449748 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.449494 4947 scope.go:122] "RemoveContainer" containerID="9088ac672a34d90115e6063e35401fc27e3e6e4e5ef995344bedfe610c88a293" Nov 03 17:37:28.459665 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.459647 4947 scope.go:122] "RemoveContainer" containerID="9088ac672a34d90115e6063e35401fc27e3e6e4e5ef995344bedfe610c88a293" Nov 03 17:37:28.459965 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:37:28.459945 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9088ac672a34d90115e6063e35401fc27e3e6e4e5ef995344bedfe610c88a293\": container with ID starting with 9088ac672a34d90115e6063e35401fc27e3e6e4e5ef995344bedfe610c88a293 not found: ID does not exist" containerID="9088ac672a34d90115e6063e35401fc27e3e6e4e5ef995344bedfe610c88a293" Nov 03 17:37:28.460020 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.459972 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9088ac672a34d90115e6063e35401fc27e3e6e4e5ef995344bedfe610c88a293"} err="failed to get container status \"9088ac672a34d90115e6063e35401fc27e3e6e4e5ef995344bedfe610c88a293\": rpc error: code = NotFound desc = could not find container \"9088ac672a34d90115e6063e35401fc27e3e6e4e5ef995344bedfe610c88a293\": container with ID starting with 9088ac672a34d90115e6063e35401fc27e3e6e4e5ef995344bedfe610c88a293 not found: ID does not exist" Nov 03 17:37:28.515384 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.515351 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84"] Nov 03 17:37:28.519068 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.519041 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-460104-openshift-default-567bb9fcf-wmf84"] Nov 03 17:37:28.565091 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:28.565057 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2bd4293-1c17-455d-9237-38edc12c9890" path="/var/lib/kubelet/pods/f2bd4293-1c17-455d-9237-38edc12c9890/volumes" Nov 03 17:37:29.948808 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:29.948763 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:37:30.131060 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.131026 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-workload-socket\") pod \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " Nov 03 17:37:30.131269 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.131067 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-workload-certs\") pod \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " Nov 03 17:37:30.131269 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.131139 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-envoy\") pod \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " Nov 03 17:37:30.131269 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.131185 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istiod-ca-cert\") pod \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " Nov 03 17:37:30.131269 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.131261 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-credential-socket\") pod \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " Nov 03 17:37:30.131520 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.131299 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-kube-api-access-4m2hh\" (UniqueName: \"kubernetes.io/projected/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-kube-api-access-4m2hh\") pod \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " Nov 03 17:37:30.131520 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.131347 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-workload-socket" pod "3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" (UID: "3e42485e-7fc4-4a72-ae33-cf3e4f2acb86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:30.131520 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.131379 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-data\") pod \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " Nov 03 17:37:30.131520 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.131420 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-podinfo\") pod \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " Nov 03 17:37:30.131520 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.131455 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-token\" (UniqueName: \"kubernetes.io/projected/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-token\") pod \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\" (UID: \"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86\") " Nov 03 17:37:30.132186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.131485 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-workload-certs" pod "3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" (UID: "3e42485e-7fc4-4a72-ae33-cf3e4f2acb86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:30.132186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.131550 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-credential-socket" pod "3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" (UID: "3e42485e-7fc4-4a72-ae33-cf3e4f2acb86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:30.132186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.131771 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-data" pod "3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" (UID: "3e42485e-7fc4-4a72-ae33-cf3e4f2acb86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:30.132186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.131859 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istiod-ca-cert" pod "3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" (UID: "3e42485e-7fc4-4a72-ae33-cf3e4f2acb86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:30.132186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.131932 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:30.132186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.131951 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:30.132186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.131965 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:30.132186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.131978 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:30.133987 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:37:30.133951 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:37:30.134357 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.134329 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-token" pod "3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" (UID: "3e42485e-7fc4-4a72-ae33-cf3e4f2acb86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:30.134451 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.134332 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-envoy" pod "3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" (UID: "3e42485e-7fc4-4a72-ae33-cf3e4f2acb86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:30.134451 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.134361 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-kube-api-access-4m2hh" pod "3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" (UID: "3e42485e-7fc4-4a72-ae33-cf3e4f2acb86"). InnerVolumeSpecName "kube-api-access-4m2hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:30.134451 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.134434 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-podinfo" pod "3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" (UID: "3e42485e-7fc4-4a72-ae33-cf3e4f2acb86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:37:30.185605 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.185557 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4"] Nov 03 17:37:30.190908 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.190879 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g"] Nov 03 17:37:30.200474 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.200449 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq"] Nov 03 17:37:30.233028 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.233001 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m2hh\" (UniqueName: \"kubernetes.io/projected/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-kube-api-access-4m2hh\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:30.233028 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.233026 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:30.233252 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.233037 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:30.233252 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.233045 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:30.233252 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.233052 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:30.462979 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.462896 4947 generic.go:356] "Generic (PLEG): container finished" podID="3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" containerID="7c1360f18cf9a93c9bf64d8ce4970168d9f9c88b99c4bafb4a0ed28e0b65b818" exitCode=0 Nov 03 17:37:30.462979 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.462956 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" Nov 03 17:37:30.463176 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.462978 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" event={"ID":"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86","Type":"ContainerDied","Data":"7c1360f18cf9a93c9bf64d8ce4970168d9f9c88b99c4bafb4a0ed28e0b65b818"} Nov 03 17:37:30.463176 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.463062 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4" event={"ID":"3e42485e-7fc4-4a72-ae33-cf3e4f2acb86","Type":"ContainerDied","Data":"d3da91d561ea3df9504090787cc2eb47faaa6bbb1eca5906dfa02682ddccf9b7"} Nov 03 17:37:30.463176 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.463085 4947 scope.go:122] "RemoveContainer" containerID="7c1360f18cf9a93c9bf64d8ce4970168d9f9c88b99c4bafb4a0ed28e0b65b818" Nov 03 17:37:30.473406 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.473387 4947 scope.go:122] "RemoveContainer" containerID="7c1360f18cf9a93c9bf64d8ce4970168d9f9c88b99c4bafb4a0ed28e0b65b818" Nov 03 17:37:30.473707 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:37:30.473672 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c1360f18cf9a93c9bf64d8ce4970168d9f9c88b99c4bafb4a0ed28e0b65b818\": container with ID starting with 7c1360f18cf9a93c9bf64d8ce4970168d9f9c88b99c4bafb4a0ed28e0b65b818 not found: ID does not exist" containerID="7c1360f18cf9a93c9bf64d8ce4970168d9f9c88b99c4bafb4a0ed28e0b65b818" Nov 03 17:37:30.473773 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.473721 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c1360f18cf9a93c9bf64d8ce4970168d9f9c88b99c4bafb4a0ed28e0b65b818"} err="failed to get container status \"7c1360f18cf9a93c9bf64d8ce4970168d9f9c88b99c4bafb4a0ed28e0b65b818\": rpc error: code = NotFound desc = could not find container \"7c1360f18cf9a93c9bf64d8ce4970168d9f9c88b99c4bafb4a0ed28e0b65b818\": container with ID starting with 7c1360f18cf9a93c9bf64d8ce4970168d9f9c88b99c4bafb4a0ed28e0b65b818 not found: ID does not exist" Nov 03 17:37:30.517861 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.517831 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4"] Nov 03 17:37:30.526222 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.526199 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-e2d835-openshift-default-6665cc57bf-kt9z4"] Nov 03 17:37:30.565327 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:30.565298 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" path="/var/lib/kubelet/pods/3e42485e-7fc4-4a72-ae33-cf3e4f2acb86/volumes" Nov 03 17:37:38.154145 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:38.154108 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:37:44.855185 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:44.855145 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l"] Nov 03 17:37:44.893383 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:44.893347 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn"] Nov 03 17:37:45.207167 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.207078 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" podUID="2f4b9526-8df4-4d0f-a9bc-34824126f4ca" containerName="main" containerID="cri-o://d0b5a2d2850a9588f753a26249a1fdf03865734f7b7ddbd81950dcff11502b0d" gracePeriod=45 Nov 03 17:37:45.211141 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.211104 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" containerID="cri-o://70536f0162e8a5b210822a81676c3484ed0aaa8a6ddedd75db0564360e3f5e8d" gracePeriod=45 Nov 03 17:37:45.220252 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.220223 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" containerID="cri-o://5c4ec8e254d836f4e5d8eb3149e8c2e53566da946789536f1ad3411969176d46" gracePeriod=45 Nov 03 17:37:45.518390 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.518361 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 17:37:45.546784 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.545809 4947 generic.go:356] "Generic (PLEG): container finished" podID="f1f96872-015d-4e32-adae-b8f9a552909e" containerID="70536f0162e8a5b210822a81676c3484ed0aaa8a6ddedd75db0564360e3f5e8d" exitCode=0 Nov 03 17:37:45.546784 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.545932 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" event={"ID":"f1f96872-015d-4e32-adae-b8f9a552909e","Type":"ContainerDied","Data":"70536f0162e8a5b210822a81676c3484ed0aaa8a6ddedd75db0564360e3f5e8d"} Nov 03 17:37:45.548434 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.548306 4947 generic.go:356] "Generic (PLEG): container finished" podID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerID="5c4ec8e254d836f4e5d8eb3149e8c2e53566da946789536f1ad3411969176d46" exitCode=0 Nov 03 17:37:45.548563 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.548496 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" event={"ID":"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a","Type":"ContainerDied","Data":"5c4ec8e254d836f4e5d8eb3149e8c2e53566da946789536f1ad3411969176d46"} Nov 03 17:37:45.550149 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.550128 4947 generic.go:356] "Generic (PLEG): container finished" podID="2f4b9526-8df4-4d0f-a9bc-34824126f4ca" containerID="d0b5a2d2850a9588f753a26249a1fdf03865734f7b7ddbd81950dcff11502b0d" exitCode=0 Nov 03 17:37:45.550294 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.550188 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" Nov 03 17:37:45.550294 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.550201 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" event={"ID":"2f4b9526-8df4-4d0f-a9bc-34824126f4ca","Type":"ContainerDied","Data":"d0b5a2d2850a9588f753a26249a1fdf03865734f7b7ddbd81950dcff11502b0d"} Nov 03 17:37:45.550294 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.550237 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4" event={"ID":"2f4b9526-8df4-4d0f-a9bc-34824126f4ca","Type":"ContainerDied","Data":"ba1030c7a530eb645a0ce631feec0a53bd68290d1fcadb3cb2f6785f7550d28f"} Nov 03 17:37:45.550294 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.550261 4947 scope.go:122] "RemoveContainer" containerID="d0b5a2d2850a9588f753a26249a1fdf03865734f7b7ddbd81950dcff11502b0d" Nov 03 17:37:45.563007 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.562774 4947 scope.go:122] "RemoveContainer" containerID="d0b5a2d2850a9588f753a26249a1fdf03865734f7b7ddbd81950dcff11502b0d" Nov 03 17:37:45.563204 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:37:45.563176 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b5a2d2850a9588f753a26249a1fdf03865734f7b7ddbd81950dcff11502b0d\": container with ID starting with d0b5a2d2850a9588f753a26249a1fdf03865734f7b7ddbd81950dcff11502b0d not found: ID does not exist" containerID="d0b5a2d2850a9588f753a26249a1fdf03865734f7b7ddbd81950dcff11502b0d" Nov 03 17:37:45.563251 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.563219 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b5a2d2850a9588f753a26249a1fdf03865734f7b7ddbd81950dcff11502b0d"} err="failed to get container status \"d0b5a2d2850a9588f753a26249a1fdf03865734f7b7ddbd81950dcff11502b0d\": rpc error: code = NotFound desc = could not find container \"d0b5a2d2850a9588f753a26249a1fdf03865734f7b7ddbd81950dcff11502b0d\": container with ID starting with d0b5a2d2850a9588f753a26249a1fdf03865734f7b7ddbd81950dcff11502b0d not found: ID does not exist" Nov 03 17:37:45.569973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.569672 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 17:37:45.576143 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.576121 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 17:37:45.597275 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.597247 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-tls-certs\") pod \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " Nov 03 17:37:45.597471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.597300 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-home\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-home\") pod \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " Nov 03 17:37:45.597471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.597328 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-dshm\") pod \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " Nov 03 17:37:45.597471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.597346 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-tmp-dir\") pod \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " Nov 03 17:37:45.597471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.597378 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-model-cache\") pod \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " Nov 03 17:37:45.597726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.597481 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-kube-api-access-sr7rn\" (UniqueName: \"kubernetes.io/projected/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-kube-api-access-sr7rn\") pod \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\" (UID: \"2f4b9526-8df4-4d0f-a9bc-34824126f4ca\") " Nov 03 17:37:45.598897 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.597840 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-tmp-dir" pod "2f4b9526-8df4-4d0f-a9bc-34824126f4ca" (UID: "2f4b9526-8df4-4d0f-a9bc-34824126f4ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:45.598897 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.597924 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-model-cache" pod "2f4b9526-8df4-4d0f-a9bc-34824126f4ca" (UID: "2f4b9526-8df4-4d0f-a9bc-34824126f4ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:45.598897 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.598433 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-home" pod "2f4b9526-8df4-4d0f-a9bc-34824126f4ca" (UID: "2f4b9526-8df4-4d0f-a9bc-34824126f4ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:45.600079 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.600052 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-tls-certs" pod "2f4b9526-8df4-4d0f-a9bc-34824126f4ca" (UID: "2f4b9526-8df4-4d0f-a9bc-34824126f4ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:45.600619 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.600597 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-dshm" pod "2f4b9526-8df4-4d0f-a9bc-34824126f4ca" (UID: "2f4b9526-8df4-4d0f-a9bc-34824126f4ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:45.600759 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.600628 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-kube-api-access-sr7rn" pod "2f4b9526-8df4-4d0f-a9bc-34824126f4ca" (UID: "2f4b9526-8df4-4d0f-a9bc-34824126f4ca"). InnerVolumeSpecName "kube-api-access-sr7rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:45.698861 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.698822 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-tls-certs\" (UniqueName: \"kubernetes.io/secret/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-tls-certs\") pod \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " Nov 03 17:37:45.699061 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.698871 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-model-cache\") pod \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " Nov 03 17:37:45.699061 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.698891 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-home\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-home\") pod \"f1f96872-015d-4e32-adae-b8f9a552909e\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " Nov 03 17:37:45.699061 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.698917 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-dshm\") pod \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " Nov 03 17:37:45.699061 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.698987 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-tmp-dir\") pod \"f1f96872-015d-4e32-adae-b8f9a552909e\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " Nov 03 17:37:45.699277 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.699160 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-model-cache" pod "faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" (UID: "faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:45.699277 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.699238 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-model-cache\") pod \"f1f96872-015d-4e32-adae-b8f9a552909e\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " Nov 03 17:37:45.699375 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.699286 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1f96872-015d-4e32-adae-b8f9a552909e-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f96872-015d-4e32-adae-b8f9a552909e-tls-certs\") pod \"f1f96872-015d-4e32-adae-b8f9a552909e\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " Nov 03 17:37:45.699375 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.699340 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-kube-api-access-9qcm9\" (UniqueName: \"kubernetes.io/projected/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-kube-api-access-9qcm9\") pod \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " Nov 03 17:37:45.699481 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.699402 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-home\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-home\") pod \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " Nov 03 17:37:45.699481 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.699428 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-dshm\") pod \"f1f96872-015d-4e32-adae-b8f9a552909e\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " Nov 03 17:37:45.699587 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.699512 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-tmp-dir\") pod \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\" (UID: \"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a\") " Nov 03 17:37:45.699634 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.699585 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1f96872-015d-4e32-adae-b8f9a552909e-kube-api-access-722bm\" (UniqueName: \"kubernetes.io/projected/f1f96872-015d-4e32-adae-b8f9a552909e-kube-api-access-722bm\") pod \"f1f96872-015d-4e32-adae-b8f9a552909e\" (UID: \"f1f96872-015d-4e32-adae-b8f9a552909e\") " Nov 03 17:37:45.700356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.699936 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-home" pod "f1f96872-015d-4e32-adae-b8f9a552909e" (UID: "f1f96872-015d-4e32-adae-b8f9a552909e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:45.700356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.700013 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.700356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.700031 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.700356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.700045 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.700356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.700056 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.700356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.700068 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.700356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.700080 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.700356 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.700093 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr7rn\" (UniqueName: \"kubernetes.io/projected/2f4b9526-8df4-4d0f-a9bc-34824126f4ca-kube-api-access-sr7rn\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.701271 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.701108 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-model-cache" pod "f1f96872-015d-4e32-adae-b8f9a552909e" (UID: "f1f96872-015d-4e32-adae-b8f9a552909e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:45.701790 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.701733 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-home" pod "faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" (UID: "faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:45.701904 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.701797 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-tmp-dir" pod "faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" (UID: "faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:45.702051 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.702008 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-tls-certs" pod "faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" (UID: "faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:45.702512 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.702450 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-dshm" pod "faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" (UID: "faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:45.702512 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.702460 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-tmp-dir" pod "f1f96872-015d-4e32-adae-b8f9a552909e" (UID: "f1f96872-015d-4e32-adae-b8f9a552909e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:45.703149 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.703115 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f96872-015d-4e32-adae-b8f9a552909e-kube-api-access-722bm" pod "f1f96872-015d-4e32-adae-b8f9a552909e" (UID: "f1f96872-015d-4e32-adae-b8f9a552909e"). InnerVolumeSpecName "kube-api-access-722bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:45.705536 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.704127 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-kube-api-access-9qcm9" pod "faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" (UID: "faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a"). InnerVolumeSpecName "kube-api-access-9qcm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:45.705536 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.704877 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-dshm" pod "f1f96872-015d-4e32-adae-b8f9a552909e" (UID: "f1f96872-015d-4e32-adae-b8f9a552909e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:37:45.705536 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.705493 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f96872-015d-4e32-adae-b8f9a552909e-tls-certs" pod "f1f96872-015d-4e32-adae-b8f9a552909e" (UID: "f1f96872-015d-4e32-adae-b8f9a552909e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:45.801570 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.801532 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-722bm\" (UniqueName: \"kubernetes.io/projected/f1f96872-015d-4e32-adae-b8f9a552909e-kube-api-access-722bm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.801570 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.801566 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.801570 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.801577 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.801873 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.801586 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.801873 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.801595 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.801873 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.801602 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.801873 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.801610 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f96872-015d-4e32-adae-b8f9a552909e-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.801873 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.801618 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qcm9\" (UniqueName: \"kubernetes.io/projected/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-kube-api-access-9qcm9\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.801873 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.801625 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.801873 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.801632 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1f96872-015d-4e32-adae-b8f9a552909e-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.801873 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.801640 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:45.897572 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.897541 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4"] Nov 03 17:37:45.904174 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:45.904150 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-491905/distinct-2-w1-491905-kserve-59dfd8b56f-ghhw4"] Nov 03 17:37:46.556586 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:46.556557 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" Nov 03 17:37:46.556821 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:46.556605 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g" event={"ID":"f1f96872-015d-4e32-adae-b8f9a552909e","Type":"ContainerDied","Data":"4eb838c03171f72ef894ac2c9ea0ffca99de8116852333fe4abbb6efc7351b08"} Nov 03 17:37:46.556821 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:46.556656 4947 scope.go:122] "RemoveContainer" containerID="70536f0162e8a5b210822a81676c3484ed0aaa8a6ddedd75db0564360e3f5e8d" Nov 03 17:37:46.558235 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:46.558214 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" Nov 03 17:37:46.567604 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:46.567524 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f4b9526-8df4-4d0f-a9bc-34824126f4ca" path="/var/lib/kubelet/pods/2f4b9526-8df4-4d0f-a9bc-34824126f4ca/volumes" Nov 03 17:37:46.568519 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:46.568142 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq" event={"ID":"faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a","Type":"ContainerDied","Data":"5c3666703ec6aa44ce8dd6949eaab7e63c31c9a72ee624b96a1afa7c5fe30411"} Nov 03 17:37:46.568519 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:46.568184 4947 scope.go:122] "RemoveContainer" containerID="5c4ec8e254d836f4e5d8eb3149e8c2e53566da946789536f1ad3411969176d46" Nov 03 17:37:46.613775 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:46.613742 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq"] Nov 03 17:37:46.626939 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:46.626908 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-491905/premium-w1-491905-kserve-6446897f6c-rs5sq"] Nov 03 17:37:46.661358 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:46.661330 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g"] Nov 03 17:37:46.664280 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:46.664258 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-491905/distinct-w1-491905-kserve-69894cf86b-swc6g"] Nov 03 17:37:48.564491 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:48.564461 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" path="/var/lib/kubelet/pods/f1f96872-015d-4e32-adae-b8f9a552909e/volumes" Nov 03 17:37:48.564904 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:48.564872 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" path="/var/lib/kubelet/pods/faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a/volumes" Nov 03 17:37:49.856145 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:49.856098 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l" podUID="182c2154-67b3-47fe-a0df-730274bde9e0" containerName="payload-processing" containerID="cri-o://7bf333998bc1f8d671d7397c93cf29a5d0141a537dde886fb2d0a65ee3b029f3" gracePeriod=25 Nov 03 17:37:49.894563 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:49.894505 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn" podUID="23ddf118-a651-4201-85a6-e11ed761cceb" containerName="payload-pre-processing" containerID="cri-o://f49181cadc635fb61be9c711bfbae99b5a6510d8e59454b9550076a717f7c164" gracePeriod=25 Nov 03 17:37:50.157744 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.151078 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l" Nov 03 17:37:50.191599 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.191576 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn" Nov 03 17:37:50.246090 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.246061 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23ddf118-a651-4201-85a6-e11ed761cceb-kube-api-access-554t5\" (UniqueName: \"kubernetes.io/projected/23ddf118-a651-4201-85a6-e11ed761cceb-kube-api-access-554t5\") pod \"23ddf118-a651-4201-85a6-e11ed761cceb\" (UID: \"23ddf118-a651-4201-85a6-e11ed761cceb\") " Nov 03 17:37:50.246257 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.246150 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182c2154-67b3-47fe-a0df-730274bde9e0-kube-api-access-bkctf\" (UniqueName: \"kubernetes.io/projected/182c2154-67b3-47fe-a0df-730274bde9e0-kube-api-access-bkctf\") pod \"182c2154-67b3-47fe-a0df-730274bde9e0\" (UID: \"182c2154-67b3-47fe-a0df-730274bde9e0\") " Nov 03 17:37:50.246257 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.246188 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23ddf118-a651-4201-85a6-e11ed761cceb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ddf118-a651-4201-85a6-e11ed761cceb-plugins-config-volume\") pod \"23ddf118-a651-4201-85a6-e11ed761cceb\" (UID: \"23ddf118-a651-4201-85a6-e11ed761cceb\") " Nov 03 17:37:50.246390 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.246291 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/182c2154-67b3-47fe-a0df-730274bde9e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/182c2154-67b3-47fe-a0df-730274bde9e0-plugins-config-volume\") pod \"182c2154-67b3-47fe-a0df-730274bde9e0\" (UID: \"182c2154-67b3-47fe-a0df-730274bde9e0\") " Nov 03 17:37:50.246664 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.246634 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ddf118-a651-4201-85a6-e11ed761cceb-plugins-config-volume" pod "23ddf118-a651-4201-85a6-e11ed761cceb" (UID: "23ddf118-a651-4201-85a6-e11ed761cceb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:50.246798 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.246657 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182c2154-67b3-47fe-a0df-730274bde9e0-plugins-config-volume" pod "182c2154-67b3-47fe-a0df-730274bde9e0" (UID: "182c2154-67b3-47fe-a0df-730274bde9e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:50.248539 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.248512 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182c2154-67b3-47fe-a0df-730274bde9e0-kube-api-access-bkctf" pod "182c2154-67b3-47fe-a0df-730274bde9e0" (UID: "182c2154-67b3-47fe-a0df-730274bde9e0"). InnerVolumeSpecName "kube-api-access-bkctf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:50.248676 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.248654 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ddf118-a651-4201-85a6-e11ed761cceb-kube-api-access-554t5" pod "23ddf118-a651-4201-85a6-e11ed761cceb" (UID: "23ddf118-a651-4201-85a6-e11ed761cceb"). InnerVolumeSpecName "kube-api-access-554t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:50.347847 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.347815 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ddf118-a651-4201-85a6-e11ed761cceb-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:50.347847 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.347840 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/182c2154-67b3-47fe-a0df-730274bde9e0-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:50.347847 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.347849 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-554t5\" (UniqueName: \"kubernetes.io/projected/23ddf118-a651-4201-85a6-e11ed761cceb-kube-api-access-554t5\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:50.348092 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.347861 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkctf\" (UniqueName: \"kubernetes.io/projected/182c2154-67b3-47fe-a0df-730274bde9e0-kube-api-access-bkctf\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:50.580118 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.580089 4947 generic.go:356] "Generic (PLEG): container finished" podID="182c2154-67b3-47fe-a0df-730274bde9e0" containerID="7bf333998bc1f8d671d7397c93cf29a5d0141a537dde886fb2d0a65ee3b029f3" exitCode=0 Nov 03 17:37:50.580245 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.580119 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l" Nov 03 17:37:50.580245 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.580168 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l" event={"ID":"182c2154-67b3-47fe-a0df-730274bde9e0","Type":"ContainerDied","Data":"7bf333998bc1f8d671d7397c93cf29a5d0141a537dde886fb2d0a65ee3b029f3"} Nov 03 17:37:50.580245 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.580220 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l" event={"ID":"182c2154-67b3-47fe-a0df-730274bde9e0","Type":"ContainerDied","Data":"f00d58c1e69383612952b30daa88d98f4ff3bf9546fe6262f071d6c992140cad"} Nov 03 17:37:50.580245 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.580242 4947 scope.go:122] "RemoveContainer" containerID="7bf333998bc1f8d671d7397c93cf29a5d0141a537dde886fb2d0a65ee3b029f3" Nov 03 17:37:50.581683 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.581655 4947 generic.go:356] "Generic (PLEG): container finished" podID="23ddf118-a651-4201-85a6-e11ed761cceb" containerID="f49181cadc635fb61be9c711bfbae99b5a6510d8e59454b9550076a717f7c164" exitCode=0 Nov 03 17:37:50.581813 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.581715 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn" event={"ID":"23ddf118-a651-4201-85a6-e11ed761cceb","Type":"ContainerDied","Data":"f49181cadc635fb61be9c711bfbae99b5a6510d8e59454b9550076a717f7c164"} Nov 03 17:37:50.581813 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.581725 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn" Nov 03 17:37:50.581813 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.581757 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn" event={"ID":"23ddf118-a651-4201-85a6-e11ed761cceb","Type":"ContainerDied","Data":"1e848cbf1783a9057bfc329dc0d93091bfa34b09a8f009023a35df8f0396dba9"} Nov 03 17:37:50.589152 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.589135 4947 scope.go:122] "RemoveContainer" containerID="7bf333998bc1f8d671d7397c93cf29a5d0141a537dde886fb2d0a65ee3b029f3" Nov 03 17:37:50.589397 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:37:50.589380 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bf333998bc1f8d671d7397c93cf29a5d0141a537dde886fb2d0a65ee3b029f3\": container with ID starting with 7bf333998bc1f8d671d7397c93cf29a5d0141a537dde886fb2d0a65ee3b029f3 not found: ID does not exist" containerID="7bf333998bc1f8d671d7397c93cf29a5d0141a537dde886fb2d0a65ee3b029f3" Nov 03 17:37:50.589453 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.589404 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bf333998bc1f8d671d7397c93cf29a5d0141a537dde886fb2d0a65ee3b029f3"} err="failed to get container status \"7bf333998bc1f8d671d7397c93cf29a5d0141a537dde886fb2d0a65ee3b029f3\": rpc error: code = NotFound desc = could not find container \"7bf333998bc1f8d671d7397c93cf29a5d0141a537dde886fb2d0a65ee3b029f3\": container with ID starting with 7bf333998bc1f8d671d7397c93cf29a5d0141a537dde886fb2d0a65ee3b029f3 not found: ID does not exist" Nov 03 17:37:50.589453 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.589419 4947 scope.go:122] "RemoveContainer" containerID="f49181cadc635fb61be9c711bfbae99b5a6510d8e59454b9550076a717f7c164" Nov 03 17:37:50.597457 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.597433 4947 scope.go:122] "RemoveContainer" containerID="f49181cadc635fb61be9c711bfbae99b5a6510d8e59454b9550076a717f7c164" Nov 03 17:37:50.597740 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:37:50.597716 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f49181cadc635fb61be9c711bfbae99b5a6510d8e59454b9550076a717f7c164\": container with ID starting with f49181cadc635fb61be9c711bfbae99b5a6510d8e59454b9550076a717f7c164 not found: ID does not exist" containerID="f49181cadc635fb61be9c711bfbae99b5a6510d8e59454b9550076a717f7c164" Nov 03 17:37:50.597830 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.597746 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f49181cadc635fb61be9c711bfbae99b5a6510d8e59454b9550076a717f7c164"} err="failed to get container status \"f49181cadc635fb61be9c711bfbae99b5a6510d8e59454b9550076a717f7c164\": rpc error: code = NotFound desc = could not find container \"f49181cadc635fb61be9c711bfbae99b5a6510d8e59454b9550076a717f7c164\": container with ID starting with f49181cadc635fb61be9c711bfbae99b5a6510d8e59454b9550076a717f7c164 not found: ID does not exist" Nov 03 17:37:50.635258 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.635224 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn"] Nov 03 17:37:50.648573 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.648549 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-491905-5dddd658cf-gclsn"] Nov 03 17:37:50.684446 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.684411 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l"] Nov 03 17:37:50.688199 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:50.688175 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-491905-7c6dddc7b5-lxp4l"] Nov 03 17:37:52.563945 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:52.563904 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="182c2154-67b3-47fe-a0df-730274bde9e0" path="/var/lib/kubelet/pods/182c2154-67b3-47fe-a0df-730274bde9e0/volumes" Nov 03 17:37:52.564357 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:52.564298 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23ddf118-a651-4201-85a6-e11ed761cceb" path="/var/lib/kubelet/pods/23ddf118-a651-4201-85a6-e11ed761cceb/volumes" Nov 03 17:37:56.747358 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:56.747323 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:37:57.972184 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:57.972150 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579"] Nov 03 17:37:57.973017 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:57.972978 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" podUID="2574a131-2be5-4179-9cb2-71980b0feeb8" containerName="maas-api" containerID="cri-o://22a4f66c66af7a0019532c8856a12793c341c10ba1ac6526f8608b3ee67f4196" gracePeriod=30 Nov 03 17:37:58.253485 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.253453 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" Nov 03 17:37:58.326344 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.326305 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2574a131-2be5-4179-9cb2-71980b0feeb8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2574a131-2be5-4179-9cb2-71980b0feeb8-openshift-service-ca\") pod \"2574a131-2be5-4179-9cb2-71980b0feeb8\" (UID: \"2574a131-2be5-4179-9cb2-71980b0feeb8\") " Nov 03 17:37:58.326531 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.326351 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2574a131-2be5-4179-9cb2-71980b0feeb8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2574a131-2be5-4179-9cb2-71980b0feeb8-maas-api-tls\") pod \"2574a131-2be5-4179-9cb2-71980b0feeb8\" (UID: \"2574a131-2be5-4179-9cb2-71980b0feeb8\") " Nov 03 17:37:58.326531 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.326412 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2574a131-2be5-4179-9cb2-71980b0feeb8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2574a131-2be5-4179-9cb2-71980b0feeb8-metrics-certs\") pod \"2574a131-2be5-4179-9cb2-71980b0feeb8\" (UID: \"2574a131-2be5-4179-9cb2-71980b0feeb8\") " Nov 03 17:37:58.326630 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.326552 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2574a131-2be5-4179-9cb2-71980b0feeb8-kube-api-access-pntn2\" (UniqueName: \"kubernetes.io/projected/2574a131-2be5-4179-9cb2-71980b0feeb8-kube-api-access-pntn2\") pod \"2574a131-2be5-4179-9cb2-71980b0feeb8\" (UID: \"2574a131-2be5-4179-9cb2-71980b0feeb8\") " Nov 03 17:37:58.326836 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.326771 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2574a131-2be5-4179-9cb2-71980b0feeb8-openshift-service-ca" pod "2574a131-2be5-4179-9cb2-71980b0feeb8" (UID: "2574a131-2be5-4179-9cb2-71980b0feeb8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:37:58.326986 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.326958 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2574a131-2be5-4179-9cb2-71980b0feeb8-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:58.328920 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.328888 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2574a131-2be5-4179-9cb2-71980b0feeb8-metrics-certs" pod "2574a131-2be5-4179-9cb2-71980b0feeb8" (UID: "2574a131-2be5-4179-9cb2-71980b0feeb8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:58.329042 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.328939 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2574a131-2be5-4179-9cb2-71980b0feeb8-maas-api-tls" pod "2574a131-2be5-4179-9cb2-71980b0feeb8" (UID: "2574a131-2be5-4179-9cb2-71980b0feeb8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:37:58.329461 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.329440 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2574a131-2be5-4179-9cb2-71980b0feeb8-kube-api-access-pntn2" pod "2574a131-2be5-4179-9cb2-71980b0feeb8" (UID: "2574a131-2be5-4179-9cb2-71980b0feeb8"). InnerVolumeSpecName "kube-api-access-pntn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:37:58.429918 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.428325 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2574a131-2be5-4179-9cb2-71980b0feeb8-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:58.429918 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.428369 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2574a131-2be5-4179-9cb2-71980b0feeb8-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:58.429918 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.428383 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pntn2\" (UniqueName: \"kubernetes.io/projected/2574a131-2be5-4179-9cb2-71980b0feeb8-kube-api-access-pntn2\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:37:58.623665 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.623631 4947 generic.go:356] "Generic (PLEG): container finished" podID="2574a131-2be5-4179-9cb2-71980b0feeb8" containerID="22a4f66c66af7a0019532c8856a12793c341c10ba1ac6526f8608b3ee67f4196" exitCode=0 Nov 03 17:37:58.623846 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.623675 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" Nov 03 17:37:58.623846 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.623684 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" event={"ID":"2574a131-2be5-4179-9cb2-71980b0feeb8","Type":"ContainerDied","Data":"22a4f66c66af7a0019532c8856a12793c341c10ba1ac6526f8608b3ee67f4196"} Nov 03 17:37:58.623846 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.623753 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579" event={"ID":"2574a131-2be5-4179-9cb2-71980b0feeb8","Type":"ContainerDied","Data":"b17bdef4c1d9ba7982dcf341961975378918a481bdbfe47a6c4582cfb49230a6"} Nov 03 17:37:58.623846 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.623778 4947 scope.go:122] "RemoveContainer" containerID="22a4f66c66af7a0019532c8856a12793c341c10ba1ac6526f8608b3ee67f4196" Nov 03 17:37:58.632916 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.632895 4947 scope.go:122] "RemoveContainer" containerID="22a4f66c66af7a0019532c8856a12793c341c10ba1ac6526f8608b3ee67f4196" Nov 03 17:37:58.633184 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:37:58.633162 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22a4f66c66af7a0019532c8856a12793c341c10ba1ac6526f8608b3ee67f4196\": container with ID starting with 22a4f66c66af7a0019532c8856a12793c341c10ba1ac6526f8608b3ee67f4196 not found: ID does not exist" containerID="22a4f66c66af7a0019532c8856a12793c341c10ba1ac6526f8608b3ee67f4196" Nov 03 17:37:58.633237 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.633192 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22a4f66c66af7a0019532c8856a12793c341c10ba1ac6526f8608b3ee67f4196"} err="failed to get container status \"22a4f66c66af7a0019532c8856a12793c341c10ba1ac6526f8608b3ee67f4196\": rpc error: code = NotFound desc = could not find container \"22a4f66c66af7a0019532c8856a12793c341c10ba1ac6526f8608b3ee67f4196\": container with ID starting with 22a4f66c66af7a0019532c8856a12793c341c10ba1ac6526f8608b3ee67f4196 not found: ID does not exist" Nov 03 17:37:58.678512 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.678482 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579"] Nov 03 17:37:58.682526 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:37:58.682500 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-491905-6c57979cd4-2l579"] Nov 03 17:38:00.129776 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:38:00.129734 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:38:00.566246 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:00.566211 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2574a131-2be5-4179-9cb2-71980b0feeb8" path="/var/lib/kubelet/pods/2574a131-2be5-4179-9cb2-71980b0feeb8/volumes" Nov 03 17:38:04.759600 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:04.759563 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:38:10.114315 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:10.114275 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf"] Nov 03 17:38:10.114881 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:10.114545 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" podUID="f30d7145-f5ba-4051-ac0b-371c2a6f2334" containerName="istio-proxy" containerID="cri-o://2288c8892517dad71363aa8c75bdd838efa698174567b6ef6e6b39f1929477dc" gracePeriod=30 Nov 03 17:38:12.384481 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:12.384451 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-t6dn2_879fa899-009b-44d6-adb2-ae2d9a82e427/limitador/0.log" Nov 03 17:38:13.571426 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:13.571394 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7d958cf575-8jkmf_d1ba5407-2f49-4ac7-9b41-157697ab2306/manager/0.log" Nov 03 17:38:14.723065 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:14.723021 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-54fd7b574c-bvhhh_07a25c94-f899-43b0-89b4-d37d1fd5a364/manager/0.log" Nov 03 17:38:15.349031 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.349006 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4grck_9f54e03f-0ad5-4368-a9eb-78fad1b70156/proxy/0.log" Nov 03 17:38:15.378525 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.378501 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 17:38:15.497263 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.497236 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-workload-certs\") pod \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " Nov 03 17:38:15.497440 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.497328 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30d7145-f5ba-4051-ac0b-371c2a6f2334-kube-api-access-hztsv\" (UniqueName: \"kubernetes.io/projected/f30d7145-f5ba-4051-ac0b-371c2a6f2334-kube-api-access-hztsv\") pod \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " Nov 03 17:38:15.497486 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.497445 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-workload-socket\") pod \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " Nov 03 17:38:15.497541 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.497487 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-envoy\") pod \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " Nov 03 17:38:15.497541 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.497526 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-data\") pod \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " Nov 03 17:38:15.497660 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.497555 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-workload-certs" pod "f30d7145-f5ba-4051-ac0b-371c2a6f2334" (UID: "f30d7145-f5ba-4051-ac0b-371c2a6f2334"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:15.497660 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.497561 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-podinfo\") pod \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " Nov 03 17:38:15.497819 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.497720 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-workload-socket" pod "f30d7145-f5ba-4051-ac0b-371c2a6f2334" (UID: "f30d7145-f5ba-4051-ac0b-371c2a6f2334"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:15.497819 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.497735 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-credential-socket\") pod \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " Nov 03 17:38:15.497819 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.497771 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istiod-ca-cert\") pod \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " Nov 03 17:38:15.497819 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.497811 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-token\" (UniqueName: \"kubernetes.io/projected/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-token\") pod \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\" (UID: \"f30d7145-f5ba-4051-ac0b-371c2a6f2334\") " Nov 03 17:38:15.498058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.497839 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-data" pod "f30d7145-f5ba-4051-ac0b-371c2a6f2334" (UID: "f30d7145-f5ba-4051-ac0b-371c2a6f2334"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:15.498058 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.498002 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-credential-socket" pod "f30d7145-f5ba-4051-ac0b-371c2a6f2334" (UID: "f30d7145-f5ba-4051-ac0b-371c2a6f2334"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:15.498471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.498106 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istiod-ca-cert" pod "f30d7145-f5ba-4051-ac0b-371c2a6f2334" (UID: "f30d7145-f5ba-4051-ac0b-371c2a6f2334"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:38:15.498471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.498279 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:38:15.498471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.498301 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:38:15.498471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.498314 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:38:15.498471 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.498326 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:38:15.499827 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.499796 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30d7145-f5ba-4051-ac0b-371c2a6f2334-kube-api-access-hztsv" pod "f30d7145-f5ba-4051-ac0b-371c2a6f2334" (UID: "f30d7145-f5ba-4051-ac0b-371c2a6f2334"). InnerVolumeSpecName "kube-api-access-hztsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:38:15.499970 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.499910 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-envoy" pod "f30d7145-f5ba-4051-ac0b-371c2a6f2334" (UID: "f30d7145-f5ba-4051-ac0b-371c2a6f2334"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:38:15.500135 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.500111 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-podinfo" pod "f30d7145-f5ba-4051-ac0b-371c2a6f2334" (UID: "f30d7145-f5ba-4051-ac0b-371c2a6f2334"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:38:15.500232 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.500144 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-token" pod "f30d7145-f5ba-4051-ac0b-371c2a6f2334" (UID: "f30d7145-f5ba-4051-ac0b-371c2a6f2334"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:38:15.599294 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.599265 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:38:15.599294 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.599291 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:38:15.599294 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.599302 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:38:15.599563 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.599310 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hztsv\" (UniqueName: \"kubernetes.io/projected/f30d7145-f5ba-4051-ac0b-371c2a6f2334-kube-api-access-hztsv\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:38:15.599563 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.599319 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f30d7145-f5ba-4051-ac0b-371c2a6f2334-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:38:15.710507 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.710472 4947 generic.go:356] "Generic (PLEG): container finished" podID="f30d7145-f5ba-4051-ac0b-371c2a6f2334" containerID="2288c8892517dad71363aa8c75bdd838efa698174567b6ef6e6b39f1929477dc" exitCode=0 Nov 03 17:38:15.710743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.710526 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" Nov 03 17:38:15.710743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.710556 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" event={"ID":"f30d7145-f5ba-4051-ac0b-371c2a6f2334","Type":"ContainerDied","Data":"2288c8892517dad71363aa8c75bdd838efa698174567b6ef6e6b39f1929477dc"} Nov 03 17:38:15.710743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.710614 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf" event={"ID":"f30d7145-f5ba-4051-ac0b-371c2a6f2334","Type":"ContainerDied","Data":"41dfb53fffee41e451f847770640bfcaed995b573f036a69757446931d8c3679"} Nov 03 17:38:15.710743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.710637 4947 scope.go:122] "RemoveContainer" containerID="2288c8892517dad71363aa8c75bdd838efa698174567b6ef6e6b39f1929477dc" Nov 03 17:38:15.720475 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.720455 4947 scope.go:122] "RemoveContainer" containerID="2288c8892517dad71363aa8c75bdd838efa698174567b6ef6e6b39f1929477dc" Nov 03 17:38:15.720750 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:38:15.720731 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2288c8892517dad71363aa8c75bdd838efa698174567b6ef6e6b39f1929477dc\": container with ID starting with 2288c8892517dad71363aa8c75bdd838efa698174567b6ef6e6b39f1929477dc not found: ID does not exist" containerID="2288c8892517dad71363aa8c75bdd838efa698174567b6ef6e6b39f1929477dc" Nov 03 17:38:15.720829 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.720762 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2288c8892517dad71363aa8c75bdd838efa698174567b6ef6e6b39f1929477dc"} err="failed to get container status \"2288c8892517dad71363aa8c75bdd838efa698174567b6ef6e6b39f1929477dc\": rpc error: code = NotFound desc = could not find container \"2288c8892517dad71363aa8c75bdd838efa698174567b6ef6e6b39f1929477dc\": container with ID starting with 2288c8892517dad71363aa8c75bdd838efa698174567b6ef6e6b39f1929477dc not found: ID does not exist" Nov 03 17:38:15.781025 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.780953 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf"] Nov 03 17:38:15.789746 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:15.789719 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-491905-openshift-default-58c75fbc9-fk6wf"] Nov 03 17:38:16.564957 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:16.564916 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f30d7145-f5ba-4051-ac0b-371c2a6f2334" path="/var/lib/kubelet/pods/f30d7145-f5ba-4051-ac0b-371c2a6f2334/volumes" Nov 03 17:38:20.422384 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422346 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk"] Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422755 4947 cpu_manager.go:414] "containerMap: removing container" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422767 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422781 4947 cpu_manager.go:414] "containerMap: removing container" podUID="a6a18780-ccc0-4d02-a76b-df4593a94e29" containerName="maas-api" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422786 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6a18780-ccc0-4d02-a76b-df4593a94e29" containerName="maas-api" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422794 4947 cpu_manager.go:414] "containerMap: removing container" podUID="b48ab6d4-f151-459b-84a6-fede7e0e6d9c" containerName="main" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422799 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b48ab6d4-f151-459b-84a6-fede7e0e6d9c" containerName="main" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422810 4947 cpu_manager.go:414] "containerMap: removing container" podUID="9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b" containerName="payload-pre-processing" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422815 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b" containerName="payload-pre-processing" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422821 4947 cpu_manager.go:414] "containerMap: removing container" podUID="cfc2a861-7b75-4723-a8b0-269ab7b28ede" containerName="payload-pre-processing" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422826 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc2a861-7b75-4723-a8b0-269ab7b28ede" containerName="payload-pre-processing" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422835 4947 cpu_manager.go:414] "containerMap: removing container" podUID="da8b428a-da45-4ad3-ab48-4f2debd415e8" containerName="payload-processing" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422839 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8b428a-da45-4ad3-ab48-4f2debd415e8" containerName="payload-processing" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422847 4947 cpu_manager.go:414] "containerMap: removing container" podUID="8f62b83b-5ad8-41aa-9e18-44cab99b41fd" containerName="maas-api" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422851 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f62b83b-5ad8-41aa-9e18-44cab99b41fd" containerName="maas-api" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422859 4947 cpu_manager.go:414] "containerMap: removing container" podUID="375a1b54-34a6-4701-b10c-dcf287987ba5" containerName="payload-pre-processing" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422864 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="375a1b54-34a6-4701-b10c-dcf287987ba5" containerName="payload-pre-processing" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422870 4947 cpu_manager.go:414] "containerMap: removing container" podUID="182c2154-67b3-47fe-a0df-730274bde9e0" containerName="payload-processing" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422874 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182c2154-67b3-47fe-a0df-730274bde9e0" containerName="payload-processing" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422880 4947 cpu_manager.go:414] "containerMap: removing container" podUID="786125e1-80ab-40fc-992d-57ddef757ddd" containerName="payload-pre-processing" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422884 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="786125e1-80ab-40fc-992d-57ddef757ddd" containerName="payload-pre-processing" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422892 4947 cpu_manager.go:414] "containerMap: removing container" podUID="3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" containerName="istio-proxy" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422897 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" containerName="istio-proxy" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422903 4947 cpu_manager.go:414] "containerMap: removing container" podUID="6739e09f-4597-4915-8400-ecc28a6e122c" containerName="maas-api" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422907 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6739e09f-4597-4915-8400-ecc28a6e122c" containerName="maas-api" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422914 4947 cpu_manager.go:414] "containerMap: removing container" podUID="2574a131-2be5-4179-9cb2-71980b0feeb8" containerName="maas-api" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422919 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2574a131-2be5-4179-9cb2-71980b0feeb8" containerName="maas-api" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422924 4947 cpu_manager.go:414] "containerMap: removing container" podUID="f30d7145-f5ba-4051-ac0b-371c2a6f2334" containerName="istio-proxy" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422929 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f30d7145-f5ba-4051-ac0b-371c2a6f2334" containerName="istio-proxy" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422935 4947 cpu_manager.go:414] "containerMap: removing container" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422940 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422946 4947 cpu_manager.go:414] "containerMap: removing container" podUID="783635c1-dfe9-4471-851a-e0b3c8ee21c5" containerName="istio-proxy" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422950 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="783635c1-dfe9-4471-851a-e0b3c8ee21c5" containerName="istio-proxy" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422956 4947 cpu_manager.go:414] "containerMap: removing container" podUID="64013461-1f82-44ce-9c7e-6dfc9d4b9159" containerName="istio-proxy" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422960 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64013461-1f82-44ce-9c7e-6dfc9d4b9159" containerName="istio-proxy" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422966 4947 cpu_manager.go:414] "containerMap: removing container" podUID="4028bf00-f9f7-44ce-9de1-4ffe922e640b" containerName="istio-proxy" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422970 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4028bf00-f9f7-44ce-9de1-4ffe922e640b" containerName="istio-proxy" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422977 4947 cpu_manager.go:414] "containerMap: removing container" podUID="23ddf118-a651-4201-85a6-e11ed761cceb" containerName="payload-pre-processing" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422982 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23ddf118-a651-4201-85a6-e11ed761cceb" containerName="payload-pre-processing" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422988 4947 cpu_manager.go:414] "containerMap: removing container" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" Nov 03 17:38:20.422963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.422993 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423027 4947 cpu_manager.go:414] "containerMap: removing container" podUID="8d41dd69-ceef-4cd2-90a3-099d4562d32f" containerName="payload-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423032 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d41dd69-ceef-4cd2-90a3-099d4562d32f" containerName="payload-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423039 4947 cpu_manager.go:414] "containerMap: removing container" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423043 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423051 4947 cpu_manager.go:414] "containerMap: removing container" podUID="2f4b9526-8df4-4d0f-a9bc-34824126f4ca" containerName="main" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423055 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4b9526-8df4-4d0f-a9bc-34824126f4ca" containerName="main" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423061 4947 cpu_manager.go:414] "containerMap: removing container" podUID="fa74cda3-6b8f-4553-ade8-0b4859783708" containerName="istio-proxy" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423066 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa74cda3-6b8f-4553-ade8-0b4859783708" containerName="istio-proxy" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423074 4947 cpu_manager.go:414] "containerMap: removing container" podUID="f2bd4293-1c17-455d-9237-38edc12c9890" containerName="istio-proxy" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423078 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2bd4293-1c17-455d-9237-38edc12c9890" containerName="istio-proxy" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423087 4947 cpu_manager.go:414] "containerMap: removing container" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423091 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423097 4947 cpu_manager.go:414] "containerMap: removing container" podUID="7bf7a98e-51f6-4ad2-b400-4715880c82d7" containerName="payload-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423103 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf7a98e-51f6-4ad2-b400-4715880c82d7" containerName="payload-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423113 4947 cpu_manager.go:414] "containerMap: removing container" podUID="976c6c9d-ae09-43d7-9858-89f5acbf5e20" containerName="maas-api" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423151 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="976c6c9d-ae09-43d7-9858-89f5acbf5e20" containerName="maas-api" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423162 4947 cpu_manager.go:414] "containerMap: removing container" podUID="95d28dd4-0aae-4f71-9bbb-24e102b03b75" containerName="payload-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423166 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d28dd4-0aae-4f71-9bbb-24e102b03b75" containerName="payload-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423173 4947 cpu_manager.go:414] "containerMap: removing container" podUID="fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b" containerName="payload-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423177 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b" containerName="payload-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423185 4947 cpu_manager.go:414] "containerMap: removing container" podUID="43b5b746-c30a-428b-95ef-7f94cc3f6fd7" containerName="payload-pre-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423191 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b5b746-c30a-428b-95ef-7f94cc3f6fd7" containerName="payload-pre-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423304 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6a18780-ccc0-4d02-a76b-df4593a94e29" containerName="maas-api" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423320 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="b48ab6d4-f151-459b-84a6-fede7e0e6d9c" containerName="main" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423330 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f64bb33-aaa3-4312-b1bd-b2fe3d067f2b" containerName="payload-pre-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423341 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc2a861-7b75-4723-a8b0-269ab7b28ede" containerName="payload-pre-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423352 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8b428a-da45-4ad3-ab48-4f2debd415e8" containerName="payload-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423360 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f62b83b-5ad8-41aa-9e18-44cab99b41fd" containerName="maas-api" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423368 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="182c2154-67b3-47fe-a0df-730274bde9e0" containerName="payload-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423377 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="375a1b54-34a6-4701-b10c-dcf287987ba5" containerName="payload-pre-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423384 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="786125e1-80ab-40fc-992d-57ddef757ddd" containerName="payload-pre-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423393 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e42485e-7fc4-4a72-ae33-cf3e4f2acb86" containerName="istio-proxy" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423400 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6739e09f-4597-4915-8400-ecc28a6e122c" containerName="maas-api" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423410 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="2574a131-2be5-4179-9cb2-71980b0feeb8" containerName="maas-api" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423419 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f30d7145-f5ba-4051-ac0b-371c2a6f2334" containerName="istio-proxy" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423427 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a3bbeb6-89f0-49af-9b1e-f5c3bf03ccdd" containerName="main" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423435 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="783635c1-dfe9-4471-851a-e0b3c8ee21c5" containerName="istio-proxy" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423443 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="64013461-1f82-44ce-9c7e-6dfc9d4b9159" containerName="istio-proxy" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423450 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="4028bf00-f9f7-44ce-9de1-4ffe922e640b" containerName="istio-proxy" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423460 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="23ddf118-a651-4201-85a6-e11ed761cceb" containerName="payload-pre-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423467 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe4b1e0-bc76-4e9b-93e9-f8a59d53c780" containerName="main" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423476 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d41dd69-ceef-4cd2-90a3-099d4562d32f" containerName="payload-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423483 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa05b2a-faca-4a3f-a3c7-c12c7ac6f12a" containerName="main" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423493 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4b9526-8df4-4d0f-a9bc-34824126f4ca" containerName="main" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423504 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa74cda3-6b8f-4553-ade8-0b4859783708" containerName="istio-proxy" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423514 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40eb248-b0bd-43ab-821c-b21ad6f8a364" containerName="main" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423521 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2bd4293-1c17-455d-9237-38edc12c9890" containerName="istio-proxy" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423529 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="976c6c9d-ae09-43d7-9858-89f5acbf5e20" containerName="maas-api" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423538 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf7a98e-51f6-4ad2-b400-4715880c82d7" containerName="payload-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423546 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d28dd4-0aae-4f71-9bbb-24e102b03b75" containerName="payload-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423554 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="fab3fb78-c9d8-4ccc-b4eb-ee4425b0121b" containerName="payload-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423563 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa74cda3-6b8f-4553-ade8-0b4859783708" containerName="istio-proxy" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423572 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b5b746-c30a-428b-95ef-7f94cc3f6fd7" containerName="payload-pre-processing" Nov 03 17:38:20.425388 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.423582 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1f96872-015d-4e32-adae-b8f9a552909e" containerName="main" Nov 03 17:38:20.429328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.428861 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.447130 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.447104 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-184babf9-openshift-default-dockercfg-pwx2m\"" Nov 03 17:38:20.462544 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.462514 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk"] Nov 03 17:38:20.543010 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.542968 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-credential-socket\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.543204 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.543015 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-podinfo\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.543204 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.543145 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksfws\" (UniqueName: \"kubernetes.io/projected/4a91fa18-074c-49e9-8d43-b7420fa9a451-kube-api-access-ksfws\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.543204 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.543185 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-workload-certs\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.543332 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.543276 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-envoy\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.543332 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.543311 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a91fa18-074c-49e9-8d43-b7420fa9a451-istiod-ca-cert\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.543332 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.543331 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-data\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.543461 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.543364 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-token\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.543461 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.543407 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-workload-socket\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.644350 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.644304 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-data\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.644650 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.644371 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-token\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.644650 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.644400 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-workload-socket\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.644650 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.644446 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-credential-socket\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.644650 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.644496 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-podinfo\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.644650 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.644560 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksfws\" (UniqueName: \"kubernetes.io/projected/4a91fa18-074c-49e9-8d43-b7420fa9a451-kube-api-access-ksfws\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.645129 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.644659 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-workload-certs\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.645129 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.644769 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-envoy\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.645129 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.644802 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-data\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.645129 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.644823 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a91fa18-074c-49e9-8d43-b7420fa9a451-istiod-ca-cert\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.645129 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.644993 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-credential-socket\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.645129 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.645109 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-workload-socket\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.645551 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.645526 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a91fa18-074c-49e9-8d43-b7420fa9a451-istiod-ca-cert\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.645551 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.645541 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-workload-certs\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.647149 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.647123 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-envoy\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.647322 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.647295 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-podinfo\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.668520 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.668497 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-token\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.669271 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.669249 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksfws\" (UniqueName: \"kubernetes.io/projected/4a91fa18-074c-49e9-8d43-b7420fa9a451-kube-api-access-ksfws\") pod \"e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:20.742476 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:20.742439 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:21.058852 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:21.058819 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk"] Nov 03 17:38:21.059602 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:38:21.059574 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a91fa18_074c_49e9_8d43_b7420fa9a451.slice/crio-5e740f4ffba2d6c2e13ce3f1131124494aed57b1dd6265282cea4d4b03f48137 WatchSource:0}: Error finding container 5e740f4ffba2d6c2e13ce3f1131124494aed57b1dd6265282cea4d4b03f48137: Status 404 returned error can't find the container with id 5e740f4ffba2d6c2e13ce3f1131124494aed57b1dd6265282cea4d4b03f48137 Nov 03 17:38:21.061692 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:21.061655 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:38:21.061828 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:21.061755 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:38:21.061828 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:21.061802 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:38:21.744580 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:21.744545 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" event={"ID":"4a91fa18-074c-49e9-8d43-b7420fa9a451","Type":"ContainerStarted","Data":"ceae27659c9d29b38a0d1e9db54ef303d72f7a9e742efe37c644fbd9958a6cbf"} Nov 03 17:38:21.744996 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:21.744602 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" event={"ID":"4a91fa18-074c-49e9-8d43-b7420fa9a451","Type":"ContainerStarted","Data":"5e740f4ffba2d6c2e13ce3f1131124494aed57b1dd6265282cea4d4b03f48137"} Nov 03 17:38:21.808276 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:21.808227 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" podStartSLOduration=1.8082129820000001 podStartE2EDuration="1.808212982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:38:21.807512045 +0000 UTC m=+5045.396512013" watchObservedRunningTime="2026-11-03 17:38:21.808212982 +0000 UTC m=+5045.397212951" Nov 03 17:38:22.743231 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:22.743191 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:22.748604 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:22.748575 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:22.749059 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:22.749037 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:22.749707 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:22.749677 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:24.829543 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:24.829511 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg"] Nov 03 17:38:24.830037 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:24.829927 4947 cpu_manager.go:414] "containerMap: removing container" podUID="fa74cda3-6b8f-4553-ade8-0b4859783708" containerName="istio-proxy" Nov 03 17:38:24.830037 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:24.829939 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa74cda3-6b8f-4553-ade8-0b4859783708" containerName="istio-proxy" Nov 03 17:38:24.833210 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:24.833190 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" Nov 03 17:38:24.858653 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:24.858624 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-184babf9-serving-cert\"" Nov 03 17:38:24.862757 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:24.862723 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg"] Nov 03 17:38:24.889186 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:24.889152 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56rg5\" (UniqueName: \"kubernetes.io/projected/04d9c974-e0da-46f9-b88f-52ab63586108-kube-api-access-56rg5\") pod \"maas-api-e2e-ait-184babf9-5556ddf654-r9lqg\" (UID: \"04d9c974-e0da-46f9-b88f-52ab63586108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" Nov 03 17:38:24.889355 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:24.889196 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04d9c974-e0da-46f9-b88f-52ab63586108-maas-api-tls\") pod \"maas-api-e2e-ait-184babf9-5556ddf654-r9lqg\" (UID: \"04d9c974-e0da-46f9-b88f-52ab63586108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" Nov 03 17:38:24.889355 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:24.889303 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d9c974-e0da-46f9-b88f-52ab63586108-metrics-certs\") pod \"maas-api-e2e-ait-184babf9-5556ddf654-r9lqg\" (UID: \"04d9c974-e0da-46f9-b88f-52ab63586108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" Nov 03 17:38:24.889489 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:24.889372 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d9c974-e0da-46f9-b88f-52ab63586108-openshift-service-ca\") pod \"maas-api-e2e-ait-184babf9-5556ddf654-r9lqg\" (UID: \"04d9c974-e0da-46f9-b88f-52ab63586108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" Nov 03 17:38:24.989944 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:24.989900 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56rg5\" (UniqueName: \"kubernetes.io/projected/04d9c974-e0da-46f9-b88f-52ab63586108-kube-api-access-56rg5\") pod \"maas-api-e2e-ait-184babf9-5556ddf654-r9lqg\" (UID: \"04d9c974-e0da-46f9-b88f-52ab63586108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" Nov 03 17:38:24.989944 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:24.989936 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04d9c974-e0da-46f9-b88f-52ab63586108-maas-api-tls\") pod \"maas-api-e2e-ait-184babf9-5556ddf654-r9lqg\" (UID: \"04d9c974-e0da-46f9-b88f-52ab63586108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" Nov 03 17:38:24.990248 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:24.989958 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d9c974-e0da-46f9-b88f-52ab63586108-metrics-certs\") pod \"maas-api-e2e-ait-184babf9-5556ddf654-r9lqg\" (UID: \"04d9c974-e0da-46f9-b88f-52ab63586108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" Nov 03 17:38:24.990248 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:24.990030 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d9c974-e0da-46f9-b88f-52ab63586108-openshift-service-ca\") pod \"maas-api-e2e-ait-184babf9-5556ddf654-r9lqg\" (UID: \"04d9c974-e0da-46f9-b88f-52ab63586108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" Nov 03 17:38:24.990826 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:24.990796 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d9c974-e0da-46f9-b88f-52ab63586108-openshift-service-ca\") pod \"maas-api-e2e-ait-184babf9-5556ddf654-r9lqg\" (UID: \"04d9c974-e0da-46f9-b88f-52ab63586108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" Nov 03 17:38:24.992610 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:24.992585 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d9c974-e0da-46f9-b88f-52ab63586108-metrics-certs\") pod \"maas-api-e2e-ait-184babf9-5556ddf654-r9lqg\" (UID: \"04d9c974-e0da-46f9-b88f-52ab63586108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" Nov 03 17:38:24.992741 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:24.992616 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04d9c974-e0da-46f9-b88f-52ab63586108-maas-api-tls\") pod \"maas-api-e2e-ait-184babf9-5556ddf654-r9lqg\" (UID: \"04d9c974-e0da-46f9-b88f-52ab63586108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" Nov 03 17:38:25.023743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:25.023684 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56rg5\" (UniqueName: \"kubernetes.io/projected/04d9c974-e0da-46f9-b88f-52ab63586108-kube-api-access-56rg5\") pod \"maas-api-e2e-ait-184babf9-5556ddf654-r9lqg\" (UID: \"04d9c974-e0da-46f9-b88f-52ab63586108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" Nov 03 17:38:25.147171 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:25.147078 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" Nov 03 17:38:25.458490 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:25.458459 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg"] Nov 03 17:38:25.459612 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:38:25.459587 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d9c974_e0da_46f9_b88f_52ab63586108.slice/crio-1a2c1014e8276fe2fe914ba048f3b8dcaa3f86ed42d6d6f04b632524b5408130 WatchSource:0}: Error finding container 1a2c1014e8276fe2fe914ba048f3b8dcaa3f86ed42d6d6f04b632524b5408130: Status 404 returned error can't find the container with id 1a2c1014e8276fe2fe914ba048f3b8dcaa3f86ed42d6d6f04b632524b5408130 Nov 03 17:38:25.461223 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:25.461202 4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:38:25.764984 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:25.764946 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" event={"ID":"04d9c974-e0da-46f9-b88f-52ab63586108","Type":"ContainerStarted","Data":"1a2c1014e8276fe2fe914ba048f3b8dcaa3f86ed42d6d6f04b632524b5408130"} Nov 03 17:38:26.770767 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:26.770727 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" event={"ID":"04d9c974-e0da-46f9-b88f-52ab63586108","Type":"ContainerStarted","Data":"a1255c23930f7c865adbb348e1cb58796c1114e0d0574ebc41c77b06c556ca53"} Nov 03 17:38:26.771230 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:26.770857 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" Nov 03 17:38:26.817609 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:26.817558 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" podStartSLOduration=2.441525558 podStartE2EDuration="2.817542876s" totalImagesPullingTime="376.017318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:38:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:38:26.816480054 +0000 UTC m=+5050.405480024" watchObservedRunningTime="2026-11-03 17:38:26.817542876 +0000 UTC m=+5050.406542846" Nov 03 17:38:28.580561 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:28.580520 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:38:30.150725 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:38:30.148273 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:38:32.781472 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:32.781442 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" Nov 03 17:38:38.086914 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:38.086025 4947 patch_prober.go:29] interesting pod/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": read tcp 10.132.0.2:35156->10.132.1.27:15021: read: connection reset by peer" start-of-body= Nov 03 17:38:38.086914 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:38.086093 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" podUID="4a91fa18-074c-49e9-8d43-b7420fa9a451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": read tcp 10.132.0.2:35156->10.132.1.27:15021: read: connection reset by peer" Nov 03 17:38:38.854904 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:38.854866 4947 generic.go:356] "Generic (PLEG): container finished" podID="4a91fa18-074c-49e9-8d43-b7420fa9a451" containerID="ceae27659c9d29b38a0d1e9db54ef303d72f7a9e742efe37c644fbd9958a6cbf" exitCode=0 Nov 03 17:38:38.855099 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:38.854939 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" event={"ID":"4a91fa18-074c-49e9-8d43-b7420fa9a451","Type":"ContainerDied","Data":"ceae27659c9d29b38a0d1e9db54ef303d72f7a9e742efe37c644fbd9958a6cbf"} Nov 03 17:38:38.855329 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:38.855313 4947 scope.go:122] "RemoveContainer" containerID="ceae27659c9d29b38a0d1e9db54ef303d72f7a9e742efe37c644fbd9958a6cbf" Nov 03 17:38:38.856628 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:38.856597 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:38:38.856714 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:38.856685 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:38:38.856761 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:38.856736 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:38:39.861374 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:39.861341 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" event={"ID":"4a91fa18-074c-49e9-8d43-b7420fa9a451","Type":"ContainerStarted","Data":"fa86644fc061a9bc524de5a9d2c46d744930593f0ac7f5d28ac6de67125438d0"} Nov 03 17:38:40.743003 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:40.742960 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:40.743319 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:40.743296 4947 patch_prober.go:29] interesting pod/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 03 17:38:40.743378 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:40.743348 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" podUID="4a91fa18-074c-49e9-8d43-b7420fa9a451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 03 17:38:41.743357 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:41.743322 4947 patch_prober.go:29] interesting pod/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 03 17:38:41.743830 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:41.743375 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" podUID="4a91fa18-074c-49e9-8d43-b7420fa9a451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 03 17:38:42.743159 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:42.743123 4947 patch_prober.go:29] interesting pod/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 03 17:38:42.743347 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:42.743174 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" podUID="4a91fa18-074c-49e9-8d43-b7420fa9a451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 03 17:38:43.743795 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:43.743755 4947 patch_prober.go:29] interesting pod/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 03 17:38:43.744235 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:43.743853 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" podUID="4a91fa18-074c-49e9-8d43-b7420fa9a451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 03 17:38:44.743994 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:44.743950 4947 patch_prober.go:29] interesting pod/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 03 17:38:44.744435 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:44.744022 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" podUID="4a91fa18-074c-49e9-8d43-b7420fa9a451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 03 17:38:45.746365 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:45.746334 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:45.746793 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:45.746662 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:45.747351 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:45.747326 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:38:51.179737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.179690 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg"] Nov 03 17:38:51.180234 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.179918 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" podUID="04d9c974-e0da-46f9-b88f-52ab63586108" containerName="maas-api" containerID="cri-o://a1255c23930f7c865adbb348e1cb58796c1114e0d0574ebc41c77b06c556ca53" gracePeriod=30 Nov 03 17:38:51.451595 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.451264 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" Nov 03 17:38:51.559870 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.559836 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04d9c974-e0da-46f9-b88f-52ab63586108-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d9c974-e0da-46f9-b88f-52ab63586108-openshift-service-ca\") pod \"04d9c974-e0da-46f9-b88f-52ab63586108\" (UID: \"04d9c974-e0da-46f9-b88f-52ab63586108\") " Nov 03 17:38:51.560056 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.559941 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d9c974-e0da-46f9-b88f-52ab63586108-kube-api-access-56rg5\" (UniqueName: \"kubernetes.io/projected/04d9c974-e0da-46f9-b88f-52ab63586108-kube-api-access-56rg5\") pod \"04d9c974-e0da-46f9-b88f-52ab63586108\" (UID: \"04d9c974-e0da-46f9-b88f-52ab63586108\") " Nov 03 17:38:51.560056 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.559982 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04d9c974-e0da-46f9-b88f-52ab63586108-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04d9c974-e0da-46f9-b88f-52ab63586108-maas-api-tls\") pod \"04d9c974-e0da-46f9-b88f-52ab63586108\" (UID: \"04d9c974-e0da-46f9-b88f-52ab63586108\") " Nov 03 17:38:51.560217 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.560080 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04d9c974-e0da-46f9-b88f-52ab63586108-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d9c974-e0da-46f9-b88f-52ab63586108-metrics-certs\") pod \"04d9c974-e0da-46f9-b88f-52ab63586108\" (UID: \"04d9c974-e0da-46f9-b88f-52ab63586108\") " Nov 03 17:38:51.560263 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.560221 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d9c974-e0da-46f9-b88f-52ab63586108-openshift-service-ca" pod "04d9c974-e0da-46f9-b88f-52ab63586108" (UID: "04d9c974-e0da-46f9-b88f-52ab63586108"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:38:51.560505 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.560480 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d9c974-e0da-46f9-b88f-52ab63586108-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:38:51.562220 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.562190 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d9c974-e0da-46f9-b88f-52ab63586108-kube-api-access-56rg5" pod "04d9c974-e0da-46f9-b88f-52ab63586108" (UID: "04d9c974-e0da-46f9-b88f-52ab63586108"). InnerVolumeSpecName "kube-api-access-56rg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:38:51.562813 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.562792 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04d9c974-e0da-46f9-b88f-52ab63586108-metrics-certs" pod "04d9c974-e0da-46f9-b88f-52ab63586108" (UID: "04d9c974-e0da-46f9-b88f-52ab63586108"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:38:51.562894 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.562813 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04d9c974-e0da-46f9-b88f-52ab63586108-maas-api-tls" pod "04d9c974-e0da-46f9-b88f-52ab63586108" (UID: "04d9c974-e0da-46f9-b88f-52ab63586108"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:38:51.661022 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.660980 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56rg5\" (UniqueName: \"kubernetes.io/projected/04d9c974-e0da-46f9-b88f-52ab63586108-kube-api-access-56rg5\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:38:51.661022 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.661010 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04d9c974-e0da-46f9-b88f-52ab63586108-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:38:51.661022 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.661019 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04d9c974-e0da-46f9-b88f-52ab63586108-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:38:51.923147 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.923109 4947 generic.go:356] "Generic (PLEG): container finished" podID="04d9c974-e0da-46f9-b88f-52ab63586108" containerID="a1255c23930f7c865adbb348e1cb58796c1114e0d0574ebc41c77b06c556ca53" exitCode=0 Nov 03 17:38:51.923334 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.923213 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" Nov 03 17:38:51.923334 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.923216 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" event={"ID":"04d9c974-e0da-46f9-b88f-52ab63586108","Type":"ContainerDied","Data":"a1255c23930f7c865adbb348e1cb58796c1114e0d0574ebc41c77b06c556ca53"} Nov 03 17:38:51.923419 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.923402 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg" event={"ID":"04d9c974-e0da-46f9-b88f-52ab63586108","Type":"ContainerDied","Data":"1a2c1014e8276fe2fe914ba048f3b8dcaa3f86ed42d6d6f04b632524b5408130"} Nov 03 17:38:51.923480 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.923432 4947 scope.go:122] "RemoveContainer" containerID="a1255c23930f7c865adbb348e1cb58796c1114e0d0574ebc41c77b06c556ca53" Nov 03 17:38:51.934853 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.934827 4947 scope.go:122] "RemoveContainer" containerID="a1255c23930f7c865adbb348e1cb58796c1114e0d0574ebc41c77b06c556ca53" Nov 03 17:38:51.935265 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:38:51.935237 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1255c23930f7c865adbb348e1cb58796c1114e0d0574ebc41c77b06c556ca53\": container with ID starting with a1255c23930f7c865adbb348e1cb58796c1114e0d0574ebc41c77b06c556ca53 not found: ID does not exist" containerID="a1255c23930f7c865adbb348e1cb58796c1114e0d0574ebc41c77b06c556ca53" Nov 03 17:38:51.935355 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.935276 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1255c23930f7c865adbb348e1cb58796c1114e0d0574ebc41c77b06c556ca53"} err="failed to get container status \"a1255c23930f7c865adbb348e1cb58796c1114e0d0574ebc41c77b06c556ca53\": rpc error: code = NotFound desc = could not find container \"a1255c23930f7c865adbb348e1cb58796c1114e0d0574ebc41c77b06c556ca53\": container with ID starting with a1255c23930f7c865adbb348e1cb58796c1114e0d0574ebc41c77b06c556ca53 not found: ID does not exist" Nov 03 17:38:51.983342 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.983316 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg"] Nov 03 17:38:51.995424 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:51.995405 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-184babf9-5556ddf654-r9lqg"] Nov 03 17:38:52.564557 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:52.564515 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04d9c974-e0da-46f9-b88f-52ab63586108" path="/var/lib/kubelet/pods/04d9c974-e0da-46f9-b88f-52ab63586108/volumes" Nov 03 17:38:53.487118 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:53.487079 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:38:55.709051 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:55.709016 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk"] Nov 03 17:38:55.709452 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:55.709283 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" podUID="4a91fa18-074c-49e9-8d43-b7420fa9a451" containerName="istio-proxy" containerID="cri-o://fa86644fc061a9bc524de5a9d2c46d744930593f0ac7f5d28ac6de67125438d0" gracePeriod=30 Nov 03 17:38:59.630213 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:59.630129 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54fd7b574c-bvhhh"] Nov 03 17:38:59.630741 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:59.630344 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-54fd7b574c-bvhhh" podUID="07a25c94-f899-43b0-89b4-d37d1fd5a364" containerName="manager" containerID="cri-o://9c6fca25f979a0e22f4484f55494ee95049938af84a7b3807e231e0362a0cbc1" gracePeriod=30 Nov 03 17:38:59.881657 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:59.881600 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd7b574c-bvhhh" Nov 03 17:38:59.962657 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:59.962621 4947 generic.go:356] "Generic (PLEG): container finished" podID="07a25c94-f899-43b0-89b4-d37d1fd5a364" containerID="9c6fca25f979a0e22f4484f55494ee95049938af84a7b3807e231e0362a0cbc1" exitCode=0 Nov 03 17:38:59.962885 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:59.962684 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd7b574c-bvhhh" Nov 03 17:38:59.962885 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:59.962732 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd7b574c-bvhhh" event={"ID":"07a25c94-f899-43b0-89b4-d37d1fd5a364","Type":"ContainerDied","Data":"9c6fca25f979a0e22f4484f55494ee95049938af84a7b3807e231e0362a0cbc1"} Nov 03 17:38:59.962885 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:59.962790 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd7b574c-bvhhh" event={"ID":"07a25c94-f899-43b0-89b4-d37d1fd5a364","Type":"ContainerDied","Data":"1d3abe78eb1198f042f9d6e3af820182118414ce89c04924f0999e621386c796"} Nov 03 17:38:59.962885 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:59.962822 4947 scope.go:122] "RemoveContainer" containerID="9c6fca25f979a0e22f4484f55494ee95049938af84a7b3807e231e0362a0cbc1" Nov 03 17:38:59.975920 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:59.975892 4947 scope.go:122] "RemoveContainer" containerID="9c6fca25f979a0e22f4484f55494ee95049938af84a7b3807e231e0362a0cbc1" Nov 03 17:38:59.976225 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:38:59.976203 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c6fca25f979a0e22f4484f55494ee95049938af84a7b3807e231e0362a0cbc1\": container with ID starting with 9c6fca25f979a0e22f4484f55494ee95049938af84a7b3807e231e0362a0cbc1 not found: ID does not exist" containerID="9c6fca25f979a0e22f4484f55494ee95049938af84a7b3807e231e0362a0cbc1" Nov 03 17:38:59.976342 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:38:59.976234 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c6fca25f979a0e22f4484f55494ee95049938af84a7b3807e231e0362a0cbc1"} err="failed to get container status \"9c6fca25f979a0e22f4484f55494ee95049938af84a7b3807e231e0362a0cbc1\": rpc error: code = NotFound desc = could not find container \"9c6fca25f979a0e22f4484f55494ee95049938af84a7b3807e231e0362a0cbc1\": container with ID starting with 9c6fca25f979a0e22f4484f55494ee95049938af84a7b3807e231e0362a0cbc1 not found: ID does not exist" Nov 03 17:39:00.040007 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.039973 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07a25c94-f899-43b0-89b4-d37d1fd5a364-webhook-cert\" (UniqueName: \"kubernetes.io/secret/07a25c94-f899-43b0-89b4-d37d1fd5a364-webhook-cert\") pod \"07a25c94-f899-43b0-89b4-d37d1fd5a364\" (UID: \"07a25c94-f899-43b0-89b4-d37d1fd5a364\") " Nov 03 17:39:00.040205 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.040042 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a25c94-f899-43b0-89b4-d37d1fd5a364-kube-api-access-v2l6w\" (UniqueName: \"kubernetes.io/projected/07a25c94-f899-43b0-89b4-d37d1fd5a364-kube-api-access-v2l6w\") pod \"07a25c94-f899-43b0-89b4-d37d1fd5a364\" (UID: \"07a25c94-f899-43b0-89b4-d37d1fd5a364\") " Nov 03 17:39:00.040205 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.040113 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07a25c94-f899-43b0-89b4-d37d1fd5a364-metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a25c94-f899-43b0-89b4-d37d1fd5a364-metrics-certs\") pod \"07a25c94-f899-43b0-89b4-d37d1fd5a364\" (UID: \"07a25c94-f899-43b0-89b4-d37d1fd5a364\") " Nov 03 17:39:00.042298 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.042261 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a25c94-f899-43b0-89b4-d37d1fd5a364-metrics-certs" pod "07a25c94-f899-43b0-89b4-d37d1fd5a364" (UID: "07a25c94-f899-43b0-89b4-d37d1fd5a364"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:39:00.042421 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.042328 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a25c94-f899-43b0-89b4-d37d1fd5a364-webhook-cert" pod "07a25c94-f899-43b0-89b4-d37d1fd5a364" (UID: "07a25c94-f899-43b0-89b4-d37d1fd5a364"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:39:00.042421 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.042359 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a25c94-f899-43b0-89b4-d37d1fd5a364-kube-api-access-v2l6w" pod "07a25c94-f899-43b0-89b4-d37d1fd5a364" (UID: "07a25c94-f899-43b0-89b4-d37d1fd5a364"). InnerVolumeSpecName "kube-api-access-v2l6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:39:00.129036 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:39:00.128985 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:39:00.141649 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.141570 4947 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07a25c94-f899-43b0-89b4-d37d1fd5a364-webhook-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:39:00.141649 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.141603 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2l6w\" (UniqueName: \"kubernetes.io/projected/07a25c94-f899-43b0-89b4-d37d1fd5a364-kube-api-access-v2l6w\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:39:00.141649 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.141617 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a25c94-f899-43b0-89b4-d37d1fd5a364-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:39:00.311332 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.311299 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54fd7b574c-bvhhh"] Nov 03 17:39:00.316245 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.316218 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-54fd7b574c-bvhhh"] Nov 03 17:39:00.563560 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.563529 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07a25c94-f899-43b0-89b4-d37d1fd5a364" path="/var/lib/kubelet/pods/07a25c94-f899-43b0-89b4-d37d1fd5a364/volumes" Nov 03 17:39:00.747972 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.747946 4947 patch_prober.go:29] interesting pod/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Nov 03 17:39:00.748307 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.748009 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" podUID="4a91fa18-074c-49e9-8d43-b7420fa9a451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Nov 03 17:39:00.963822 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.963797 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:39:00.969158 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.969127 4947 generic.go:356] "Generic (PLEG): container finished" podID="4a91fa18-074c-49e9-8d43-b7420fa9a451" containerID="fa86644fc061a9bc524de5a9d2c46d744930593f0ac7f5d28ac6de67125438d0" exitCode=0 Nov 03 17:39:00.969312 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.969178 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" Nov 03 17:39:00.969312 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.969220 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" event={"ID":"4a91fa18-074c-49e9-8d43-b7420fa9a451","Type":"ContainerDied","Data":"fa86644fc061a9bc524de5a9d2c46d744930593f0ac7f5d28ac6de67125438d0"} Nov 03 17:39:00.969312 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.969283 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk" event={"ID":"4a91fa18-074c-49e9-8d43-b7420fa9a451","Type":"ContainerDied","Data":"5e740f4ffba2d6c2e13ce3f1131124494aed57b1dd6265282cea4d4b03f48137"} Nov 03 17:39:00.969497 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.969313 4947 scope.go:122] "RemoveContainer" containerID="fa86644fc061a9bc524de5a9d2c46d744930593f0ac7f5d28ac6de67125438d0" Nov 03 17:39:00.979503 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.979483 4947 scope.go:122] "RemoveContainer" containerID="ceae27659c9d29b38a0d1e9db54ef303d72f7a9e742efe37c644fbd9958a6cbf" Nov 03 17:39:00.987681 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.987659 4947 scope.go:122] "RemoveContainer" containerID="fa86644fc061a9bc524de5a9d2c46d744930593f0ac7f5d28ac6de67125438d0" Nov 03 17:39:00.988014 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:39:00.987991 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa86644fc061a9bc524de5a9d2c46d744930593f0ac7f5d28ac6de67125438d0\": container with ID starting with fa86644fc061a9bc524de5a9d2c46d744930593f0ac7f5d28ac6de67125438d0 not found: ID does not exist" containerID="fa86644fc061a9bc524de5a9d2c46d744930593f0ac7f5d28ac6de67125438d0" Nov 03 17:39:00.988112 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.988021 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa86644fc061a9bc524de5a9d2c46d744930593f0ac7f5d28ac6de67125438d0"} err="failed to get container status \"fa86644fc061a9bc524de5a9d2c46d744930593f0ac7f5d28ac6de67125438d0\": rpc error: code = NotFound desc = could not find container \"fa86644fc061a9bc524de5a9d2c46d744930593f0ac7f5d28ac6de67125438d0\": container with ID starting with fa86644fc061a9bc524de5a9d2c46d744930593f0ac7f5d28ac6de67125438d0 not found: ID does not exist" Nov 03 17:39:00.988112 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.988039 4947 scope.go:122] "RemoveContainer" containerID="ceae27659c9d29b38a0d1e9db54ef303d72f7a9e742efe37c644fbd9958a6cbf" Nov 03 17:39:00.988274 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:39:00.988228 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceae27659c9d29b38a0d1e9db54ef303d72f7a9e742efe37c644fbd9958a6cbf\": container with ID starting with ceae27659c9d29b38a0d1e9db54ef303d72f7a9e742efe37c644fbd9958a6cbf not found: ID does not exist" containerID="ceae27659c9d29b38a0d1e9db54ef303d72f7a9e742efe37c644fbd9958a6cbf" Nov 03 17:39:00.988274 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:00.988246 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceae27659c9d29b38a0d1e9db54ef303d72f7a9e742efe37c644fbd9958a6cbf"} err="failed to get container status \"ceae27659c9d29b38a0d1e9db54ef303d72f7a9e742efe37c644fbd9958a6cbf\": rpc error: code = NotFound desc = could not find container \"ceae27659c9d29b38a0d1e9db54ef303d72f7a9e742efe37c644fbd9958a6cbf\": container with ID starting with ceae27659c9d29b38a0d1e9db54ef303d72f7a9e742efe37c644fbd9958a6cbf not found: ID does not exist" Nov 03 17:39:01.013455 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.013406 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-54fd7b574c-ntbv5"] Nov 03 17:39:01.013881 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.013861 4947 cpu_manager.go:414] "containerMap: removing container" podUID="07a25c94-f899-43b0-89b4-d37d1fd5a364" containerName="manager" Nov 03 17:39:01.013881 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.013880 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a25c94-f899-43b0-89b4-d37d1fd5a364" containerName="manager" Nov 03 17:39:01.014085 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.013903 4947 cpu_manager.go:414] "containerMap: removing container" podUID="04d9c974-e0da-46f9-b88f-52ab63586108" containerName="maas-api" Nov 03 17:39:01.014085 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.013908 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d9c974-e0da-46f9-b88f-52ab63586108" containerName="maas-api" Nov 03 17:39:01.014085 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.013921 4947 cpu_manager.go:414] "containerMap: removing container" podUID="4a91fa18-074c-49e9-8d43-b7420fa9a451" containerName="istio-proxy" Nov 03 17:39:01.014085 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.013927 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a91fa18-074c-49e9-8d43-b7420fa9a451" containerName="istio-proxy" Nov 03 17:39:01.014085 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.013940 4947 cpu_manager.go:414] "containerMap: removing container" podUID="4a91fa18-074c-49e9-8d43-b7420fa9a451" containerName="istio-proxy" Nov 03 17:39:01.014085 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.013944 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a91fa18-074c-49e9-8d43-b7420fa9a451" containerName="istio-proxy" Nov 03 17:39:01.014085 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.014005 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a25c94-f899-43b0-89b4-d37d1fd5a364" containerName="manager" Nov 03 17:39:01.014085 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.014016 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d9c974-e0da-46f9-b88f-52ab63586108" containerName="maas-api" Nov 03 17:39:01.014085 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.014023 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a91fa18-074c-49e9-8d43-b7420fa9a451" containerName="istio-proxy" Nov 03 17:39:01.014085 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.014032 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a91fa18-074c-49e9-8d43-b7420fa9a451" containerName="istio-proxy" Nov 03 17:39:01.018298 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.018274 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd7b574c-ntbv5" Nov 03 17:39:01.035500 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.035476 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 17:39:01.037532 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.037478 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sgj9n\"" Nov 03 17:39:01.037532 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.037503 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 17:39:01.037532 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.037485 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 17:39:01.045935 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.045911 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54fd7b574c-ntbv5"] Nov 03 17:39:01.050160 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.050134 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-workload-certs\") pod \"4a91fa18-074c-49e9-8d43-b7420fa9a451\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " Nov 03 17:39:01.050312 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.050181 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-token\" (UniqueName: \"kubernetes.io/projected/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-token\") pod \"4a91fa18-074c-49e9-8d43-b7420fa9a451\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " Nov 03 17:39:01.050312 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.050216 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-podinfo\") pod \"4a91fa18-074c-49e9-8d43-b7420fa9a451\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " Nov 03 17:39:01.050453 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.050308 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a91fa18-074c-49e9-8d43-b7420fa9a451-kube-api-access-ksfws\" (UniqueName: \"kubernetes.io/projected/4a91fa18-074c-49e9-8d43-b7420fa9a451-kube-api-access-ksfws\") pod \"4a91fa18-074c-49e9-8d43-b7420fa9a451\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " Nov 03 17:39:01.050453 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.050361 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-workload-socket\") pod \"4a91fa18-074c-49e9-8d43-b7420fa9a451\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " Nov 03 17:39:01.050453 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.050396 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-envoy\") pod \"4a91fa18-074c-49e9-8d43-b7420fa9a451\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " Nov 03 17:39:01.050453 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.050428 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-credential-socket\") pod \"4a91fa18-074c-49e9-8d43-b7420fa9a451\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " Nov 03 17:39:01.050720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.050467 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a91fa18-074c-49e9-8d43-b7420fa9a451-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a91fa18-074c-49e9-8d43-b7420fa9a451-istiod-ca-cert\") pod \"4a91fa18-074c-49e9-8d43-b7420fa9a451\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " Nov 03 17:39:01.050720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.050473 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-workload-certs" pod "4a91fa18-074c-49e9-8d43-b7420fa9a451" (UID: "4a91fa18-074c-49e9-8d43-b7420fa9a451"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:39:01.050720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.050586 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-data\") pod \"4a91fa18-074c-49e9-8d43-b7420fa9a451\" (UID: \"4a91fa18-074c-49e9-8d43-b7420fa9a451\") " Nov 03 17:39:01.050720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.050663 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-workload-socket" pod "4a91fa18-074c-49e9-8d43-b7420fa9a451" (UID: "4a91fa18-074c-49e9-8d43-b7420fa9a451"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:39:01.050972 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.050890 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-credential-socket" pod "4a91fa18-074c-49e9-8d43-b7420fa9a451" (UID: "4a91fa18-074c-49e9-8d43-b7420fa9a451"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:39:01.051043 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.051004 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:39:01.051043 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.051021 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:39:01.051043 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.051035 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:39:01.051333 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.051293 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-data" pod "4a91fa18-074c-49e9-8d43-b7420fa9a451" (UID: "4a91fa18-074c-49e9-8d43-b7420fa9a451"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:39:01.051422 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.051388 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a91fa18-074c-49e9-8d43-b7420fa9a451-istiod-ca-cert" pod "4a91fa18-074c-49e9-8d43-b7420fa9a451" (UID: "4a91fa18-074c-49e9-8d43-b7420fa9a451"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:39:01.076716 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.053452 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-podinfo" pod "4a91fa18-074c-49e9-8d43-b7420fa9a451" (UID: "4a91fa18-074c-49e9-8d43-b7420fa9a451"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:39:01.076716 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.053563 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-envoy" pod "4a91fa18-074c-49e9-8d43-b7420fa9a451" (UID: "4a91fa18-074c-49e9-8d43-b7420fa9a451"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:39:01.076716 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.053768 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-token" pod "4a91fa18-074c-49e9-8d43-b7420fa9a451" (UID: "4a91fa18-074c-49e9-8d43-b7420fa9a451"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:39:01.076716 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.054777 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a91fa18-074c-49e9-8d43-b7420fa9a451-kube-api-access-ksfws" pod "4a91fa18-074c-49e9-8d43-b7420fa9a451" (UID: "4a91fa18-074c-49e9-8d43-b7420fa9a451"). InnerVolumeSpecName "kube-api-access-ksfws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:39:01.151620 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.151575 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6gh2\" (UniqueName: \"kubernetes.io/projected/cfb3dd1f-8e35-4520-b336-61ba4016aafa-kube-api-access-x6gh2\") pod \"maas-controller-54fd7b574c-ntbv5\" (UID: \"cfb3dd1f-8e35-4520-b336-61ba4016aafa\") " pod="opendatahub/maas-controller-54fd7b574c-ntbv5" Nov 03 17:39:01.151843 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.151630 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cfb3dd1f-8e35-4520-b336-61ba4016aafa-webhook-cert\") pod \"maas-controller-54fd7b574c-ntbv5\" (UID: \"cfb3dd1f-8e35-4520-b336-61ba4016aafa\") " pod="opendatahub/maas-controller-54fd7b574c-ntbv5" Nov 03 17:39:01.151843 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.151774 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb3dd1f-8e35-4520-b336-61ba4016aafa-metrics-certs\") pod \"maas-controller-54fd7b574c-ntbv5\" (UID: \"cfb3dd1f-8e35-4520-b336-61ba4016aafa\") " pod="opendatahub/maas-controller-54fd7b574c-ntbv5" Nov 03 17:39:01.151843 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.151834 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:39:01.152064 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.151873 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksfws\" (UniqueName: \"kubernetes.io/projected/4a91fa18-074c-49e9-8d43-b7420fa9a451-kube-api-access-ksfws\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:39:01.152064 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.151887 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:39:01.152064 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.151897 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a91fa18-074c-49e9-8d43-b7420fa9a451-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:39:01.152064 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.151906 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:39:01.152064 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.151914 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a91fa18-074c-49e9-8d43-b7420fa9a451-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:39:01.253150 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.253097 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cfb3dd1f-8e35-4520-b336-61ba4016aafa-webhook-cert\") pod \"maas-controller-54fd7b574c-ntbv5\" (UID: \"cfb3dd1f-8e35-4520-b336-61ba4016aafa\") " pod="opendatahub/maas-controller-54fd7b574c-ntbv5" Nov 03 17:39:01.253376 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.253229 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb3dd1f-8e35-4520-b336-61ba4016aafa-metrics-certs\") pod \"maas-controller-54fd7b574c-ntbv5\" (UID: \"cfb3dd1f-8e35-4520-b336-61ba4016aafa\") " pod="opendatahub/maas-controller-54fd7b574c-ntbv5" Nov 03 17:39:01.253376 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.253288 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6gh2\" (UniqueName: \"kubernetes.io/projected/cfb3dd1f-8e35-4520-b336-61ba4016aafa-kube-api-access-x6gh2\") pod \"maas-controller-54fd7b574c-ntbv5\" (UID: \"cfb3dd1f-8e35-4520-b336-61ba4016aafa\") " pod="opendatahub/maas-controller-54fd7b574c-ntbv5" Nov 03 17:39:01.255777 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.255753 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cfb3dd1f-8e35-4520-b336-61ba4016aafa-webhook-cert\") pod \"maas-controller-54fd7b574c-ntbv5\" (UID: \"cfb3dd1f-8e35-4520-b336-61ba4016aafa\") " pod="opendatahub/maas-controller-54fd7b574c-ntbv5" Nov 03 17:39:01.255940 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.255789 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb3dd1f-8e35-4520-b336-61ba4016aafa-metrics-certs\") pod \"maas-controller-54fd7b574c-ntbv5\" (UID: \"cfb3dd1f-8e35-4520-b336-61ba4016aafa\") " pod="opendatahub/maas-controller-54fd7b574c-ntbv5" Nov 03 17:39:01.281585 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.281560 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6gh2\" (UniqueName: \"kubernetes.io/projected/cfb3dd1f-8e35-4520-b336-61ba4016aafa-kube-api-access-x6gh2\") pod \"maas-controller-54fd7b574c-ntbv5\" (UID: \"cfb3dd1f-8e35-4520-b336-61ba4016aafa\") " pod="opendatahub/maas-controller-54fd7b574c-ntbv5" Nov 03 17:39:01.318717 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.318653 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk"] Nov 03 17:39:01.324743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.324710 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-184babf9-openshift-default-5cbbdf768c-gxtwk"] Nov 03 17:39:01.337181 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.337088 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd7b574c-ntbv5" Nov 03 17:39:01.647174 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.647141 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54fd7b574c-ntbv5"] Nov 03 17:39:01.648827 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:39:01.648789 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb3dd1f_8e35_4520_b336_61ba4016aafa.slice/crio-505035572e242122783d4644c1d3de87d744392da99fed81dc8b06ddb4022c43 WatchSource:0}: Error finding container 505035572e242122783d4644c1d3de87d744392da99fed81dc8b06ddb4022c43: Status 404 returned error can't find the container with id 505035572e242122783d4644c1d3de87d744392da99fed81dc8b06ddb4022c43 Nov 03 17:39:01.976119 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:01.976024 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd7b574c-ntbv5" event={"ID":"cfb3dd1f-8e35-4520-b336-61ba4016aafa","Type":"ContainerStarted","Data":"505035572e242122783d4644c1d3de87d744392da99fed81dc8b06ddb4022c43"} Nov 03 17:39:02.567434 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:02.567403 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a91fa18-074c-49e9-8d43-b7420fa9a451" path="/var/lib/kubelet/pods/4a91fa18-074c-49e9-8d43-b7420fa9a451/volumes" Nov 03 17:39:02.983339 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:02.983241 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd7b574c-ntbv5" event={"ID":"cfb3dd1f-8e35-4520-b336-61ba4016aafa","Type":"ContainerStarted","Data":"99f9c1896cf235e74f8dbf83bfcba3ff7bf99327ceec3526fe6df2b060ad464f"} Nov 03 17:39:02.983339 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:02.983337 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-54fd7b574c-ntbv5" Nov 03 17:39:03.033587 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:03.033518 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-54fd7b574c-ntbv5" podStartSLOduration=2.602668231 podStartE2EDuration="3.033494864s" totalImagesPullingTime="430.826633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:39:03.030244215 +0000 UTC m=+5086.619244187" watchObservedRunningTime="2026-11-03 17:39:03.033494864 +0000 UTC m=+5086.622494835" Nov 03 17:39:13.991177 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:13.991145 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-54fd7b574c-ntbv5" Nov 03 17:39:17.387515 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:17.387481 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:39:17.394149 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:17.394123 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:39:20.646201 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.646167 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95"] Nov 03 17:39:20.651422 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.651399 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.684891 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.684857 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95"] Nov 03 17:39:20.746379 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.746337 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404b1437-2e7e-49c9-a538-23399eeb21eb-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.746379 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.746385 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404b1437-2e7e-49c9-a538-23399eeb21eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.746624 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.746461 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj4pl\" (UniqueName: \"kubernetes.io/projected/404b1437-2e7e-49c9-a538-23399eeb21eb-kube-api-access-gj4pl\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.746624 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.746598 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404b1437-2e7e-49c9-a538-23399eeb21eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.746721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.746634 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404b1437-2e7e-49c9-a538-23399eeb21eb-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.746721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.746654 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404b1437-2e7e-49c9-a538-23399eeb21eb-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.746721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.746671 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404b1437-2e7e-49c9-a538-23399eeb21eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.746721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.746686 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/404b1437-2e7e-49c9-a538-23399eeb21eb-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.746876 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.746743 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404b1437-2e7e-49c9-a538-23399eeb21eb-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.847625 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.847589 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404b1437-2e7e-49c9-a538-23399eeb21eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.847625 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.847629 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404b1437-2e7e-49c9-a538-23399eeb21eb-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.847949 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.847647 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404b1437-2e7e-49c9-a538-23399eeb21eb-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.847949 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.847664 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404b1437-2e7e-49c9-a538-23399eeb21eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.847949 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.847759 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/404b1437-2e7e-49c9-a538-23399eeb21eb-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.847949 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.847795 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404b1437-2e7e-49c9-a538-23399eeb21eb-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.847949 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.847875 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404b1437-2e7e-49c9-a538-23399eeb21eb-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.847949 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.847906 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404b1437-2e7e-49c9-a538-23399eeb21eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.847949 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.847934 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj4pl\" (UniqueName: \"kubernetes.io/projected/404b1437-2e7e-49c9-a538-23399eeb21eb-kube-api-access-gj4pl\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.848403 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.848108 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/404b1437-2e7e-49c9-a538-23399eeb21eb-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.848403 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.848133 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/404b1437-2e7e-49c9-a538-23399eeb21eb-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.848482 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.848446 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/404b1437-2e7e-49c9-a538-23399eeb21eb-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.848482 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.848472 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/404b1437-2e7e-49c9-a538-23399eeb21eb-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.848550 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.848478 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/404b1437-2e7e-49c9-a538-23399eeb21eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.850297 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.850279 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/404b1437-2e7e-49c9-a538-23399eeb21eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.850518 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.850499 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/404b1437-2e7e-49c9-a538-23399eeb21eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.874264 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.874233 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/404b1437-2e7e-49c9-a538-23399eeb21eb-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.874264 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.874257 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj4pl\" (UniqueName: \"kubernetes.io/projected/404b1437-2e7e-49c9-a538-23399eeb21eb-kube-api-access-gj4pl\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-6lq95\" (UID: \"404b1437-2e7e-49c9-a538-23399eeb21eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:20.963550 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:20.963450 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:21.281318 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:21.281278 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95"] Nov 03 17:39:21.281982 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:39:21.281955 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404b1437_2e7e_49c9_a538_23399eeb21eb.slice/crio-e01b651e2c3ef9259824886032b1e13dfd6195ca39792adabb3ce7d436919529 WatchSource:0}: Error finding container e01b651e2c3ef9259824886032b1e13dfd6195ca39792adabb3ce7d436919529: Status 404 returned error can't find the container with id e01b651e2c3ef9259824886032b1e13dfd6195ca39792adabb3ce7d436919529 Nov 03 17:39:21.284142 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:21.284111 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:39:21.284217 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:21.284180 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:39:21.284217 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:21.284207 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:39:22.072874 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:22.072833 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" event={"ID":"404b1437-2e7e-49c9-a538-23399eeb21eb","Type":"ContainerStarted","Data":"7232bd76a45971490e6152e06c6d40aeecc3d8d3918b670b538bca8c3ed489fd"} Nov 03 17:39:22.073296 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:22.072920 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" event={"ID":"404b1437-2e7e-49c9-a538-23399eeb21eb","Type":"ContainerStarted","Data":"e01b651e2c3ef9259824886032b1e13dfd6195ca39792adabb3ce7d436919529"} Nov 03 17:39:22.142144 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:22.142084 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" podStartSLOduration=2.142070046 podStartE2EDuration="2.142070046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:39:22.140358094 +0000 UTC m=+5105.729358063" watchObservedRunningTime="2026-11-03 17:39:22.142070046 +0000 UTC m=+5105.731070015" Nov 03 17:39:22.964368 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:22.964335 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:22.966114 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:22.966081 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6lq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 17:39:22.966244 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:22.966137 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" podUID="404b1437-2e7e-49c9-a538-23399eeb21eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 17:39:23.964550 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:23.964512 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6lq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 17:39:23.964992 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:23.964587 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" podUID="404b1437-2e7e-49c9-a538-23399eeb21eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 17:39:24.964475 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:24.964434 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6lq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 17:39:24.964670 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:24.964501 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" podUID="404b1437-2e7e-49c9-a538-23399eeb21eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 17:39:25.964041 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:25.964004 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6lq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 17:39:25.964248 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:25.964072 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" podUID="404b1437-2e7e-49c9-a538-23399eeb21eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 17:39:26.964134 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:26.964098 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6lq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 17:39:26.964561 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:26.964164 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" podUID="404b1437-2e7e-49c9-a538-23399eeb21eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 17:39:27.964172 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:27.964139 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6lq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Nov 03 17:39:27.964601 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:27.964194 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" podUID="404b1437-2e7e-49c9-a538-23399eeb21eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Nov 03 17:39:29.964453 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:29.964414 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6lq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:39:29.965025 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:29.964481 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" podUID="404b1437-2e7e-49c9-a538-23399eeb21eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:39:30.133130 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:39:30.133090 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:39:31.005884 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:31.005843 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6lq95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:39:31.006405 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:31.005903 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" podUID="404b1437-2e7e-49c9-a538-23399eeb21eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:39:31.639758 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:31.639726 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:31.640454 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:31.640432 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:31.641082 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:31.641054 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" Nov 03 17:39:34.115145 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:34.115110 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54fd7b574c-ntbv5"] Nov 03 17:39:34.115604 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:34.115309 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-54fd7b574c-ntbv5" podUID="cfb3dd1f-8e35-4520-b336-61ba4016aafa" containerName="manager" containerID="cri-o://99f9c1896cf235e74f8dbf83bfcba3ff7bf99327ceec3526fe6df2b060ad464f" gracePeriod=30 Nov 03 17:39:34.376202 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:34.376135 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd7b574c-ntbv5" Nov 03 17:39:34.504152 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:34.504120 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb3dd1f-8e35-4520-b336-61ba4016aafa-webhook-cert\" (UniqueName: \"kubernetes.io/secret/cfb3dd1f-8e35-4520-b336-61ba4016aafa-webhook-cert\") pod \"cfb3dd1f-8e35-4520-b336-61ba4016aafa\" (UID: \"cfb3dd1f-8e35-4520-b336-61ba4016aafa\") " Nov 03 17:39:34.504373 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:34.504177 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb3dd1f-8e35-4520-b336-61ba4016aafa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb3dd1f-8e35-4520-b336-61ba4016aafa-metrics-certs\") pod \"cfb3dd1f-8e35-4520-b336-61ba4016aafa\" (UID: \"cfb3dd1f-8e35-4520-b336-61ba4016aafa\") " Nov 03 17:39:34.504373 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:34.504208 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb3dd1f-8e35-4520-b336-61ba4016aafa-kube-api-access-x6gh2\" (UniqueName: \"kubernetes.io/projected/cfb3dd1f-8e35-4520-b336-61ba4016aafa-kube-api-access-x6gh2\") pod \"cfb3dd1f-8e35-4520-b336-61ba4016aafa\" (UID: \"cfb3dd1f-8e35-4520-b336-61ba4016aafa\") " Nov 03 17:39:34.506495 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:34.506464 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb3dd1f-8e35-4520-b336-61ba4016aafa-webhook-cert" pod "cfb3dd1f-8e35-4520-b336-61ba4016aafa" (UID: "cfb3dd1f-8e35-4520-b336-61ba4016aafa"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:39:34.506620 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:34.506529 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb3dd1f-8e35-4520-b336-61ba4016aafa-kube-api-access-x6gh2" pod "cfb3dd1f-8e35-4520-b336-61ba4016aafa" (UID: "cfb3dd1f-8e35-4520-b336-61ba4016aafa"). InnerVolumeSpecName "kube-api-access-x6gh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:39:34.506620 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:34.506566 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb3dd1f-8e35-4520-b336-61ba4016aafa-metrics-certs" pod "cfb3dd1f-8e35-4520-b336-61ba4016aafa" (UID: "cfb3dd1f-8e35-4520-b336-61ba4016aafa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:39:34.605226 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:34.605195 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfb3dd1f-8e35-4520-b336-61ba4016aafa-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:39:34.605226 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:34.605223 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6gh2\" (UniqueName: \"kubernetes.io/projected/cfb3dd1f-8e35-4520-b336-61ba4016aafa-kube-api-access-x6gh2\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:39:34.605226 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:34.605233 4947 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cfb3dd1f-8e35-4520-b336-61ba4016aafa-webhook-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:39:35.135338 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.135299 4947 generic.go:356] "Generic (PLEG): container finished" podID="cfb3dd1f-8e35-4520-b336-61ba4016aafa" containerID="99f9c1896cf235e74f8dbf83bfcba3ff7bf99327ceec3526fe6df2b060ad464f" exitCode=0 Nov 03 17:39:35.135808 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.135348 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd7b574c-ntbv5" Nov 03 17:39:35.135808 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.135384 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd7b574c-ntbv5" event={"ID":"cfb3dd1f-8e35-4520-b336-61ba4016aafa","Type":"ContainerDied","Data":"99f9c1896cf235e74f8dbf83bfcba3ff7bf99327ceec3526fe6df2b060ad464f"} Nov 03 17:39:35.135808 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.135435 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd7b574c-ntbv5" event={"ID":"cfb3dd1f-8e35-4520-b336-61ba4016aafa","Type":"ContainerDied","Data":"505035572e242122783d4644c1d3de87d744392da99fed81dc8b06ddb4022c43"} Nov 03 17:39:35.135808 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.135456 4947 scope.go:122] "RemoveContainer" containerID="99f9c1896cf235e74f8dbf83bfcba3ff7bf99327ceec3526fe6df2b060ad464f" Nov 03 17:39:35.144243 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.144223 4947 scope.go:122] "RemoveContainer" containerID="99f9c1896cf235e74f8dbf83bfcba3ff7bf99327ceec3526fe6df2b060ad464f" Nov 03 17:39:35.144490 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:39:35.144462 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99f9c1896cf235e74f8dbf83bfcba3ff7bf99327ceec3526fe6df2b060ad464f\": container with ID starting with 99f9c1896cf235e74f8dbf83bfcba3ff7bf99327ceec3526fe6df2b060ad464f not found: ID does not exist" containerID="99f9c1896cf235e74f8dbf83bfcba3ff7bf99327ceec3526fe6df2b060ad464f" Nov 03 17:39:35.144550 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.144488 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99f9c1896cf235e74f8dbf83bfcba3ff7bf99327ceec3526fe6df2b060ad464f"} err="failed to get container status \"99f9c1896cf235e74f8dbf83bfcba3ff7bf99327ceec3526fe6df2b060ad464f\": rpc error: code = NotFound desc = could not find container \"99f9c1896cf235e74f8dbf83bfcba3ff7bf99327ceec3526fe6df2b060ad464f\": container with ID starting with 99f9c1896cf235e74f8dbf83bfcba3ff7bf99327ceec3526fe6df2b060ad464f not found: ID does not exist" Nov 03 17:39:35.190477 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.190453 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-54fd7b574c-ntbv5"] Nov 03 17:39:35.203819 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.203796 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-54fd7b574c-ntbv5"] Nov 03 17:39:35.705909 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.705875 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62"] Nov 03 17:39:35.706321 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.706308 4947 cpu_manager.go:414] "containerMap: removing container" podUID="cfb3dd1f-8e35-4520-b336-61ba4016aafa" containerName="manager" Nov 03 17:39:35.706382 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.706324 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb3dd1f-8e35-4520-b336-61ba4016aafa" containerName="manager" Nov 03 17:39:35.706437 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.706399 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb3dd1f-8e35-4520-b336-61ba4016aafa" containerName="manager" Nov 03 17:39:35.711429 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.711403 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.735514 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.735486 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62"] Nov 03 17:39:35.818244 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.818203 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37849d8a-115b-44a5-bea8-539a92f53653-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.818436 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.818260 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37849d8a-115b-44a5-bea8-539a92f53653-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.818436 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.818343 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37849d8a-115b-44a5-bea8-539a92f53653-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.818436 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.818394 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37849d8a-115b-44a5-bea8-539a92f53653-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.818576 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.818454 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37849d8a-115b-44a5-bea8-539a92f53653-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.818576 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.818483 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37849d8a-115b-44a5-bea8-539a92f53653-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.818576 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.818501 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37849d8a-115b-44a5-bea8-539a92f53653-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.818576 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.818522 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pggkd\" (UniqueName: \"kubernetes.io/projected/37849d8a-115b-44a5-bea8-539a92f53653-kube-api-access-pggkd\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.818742 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.818689 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37849d8a-115b-44a5-bea8-539a92f53653-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.919939 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.919902 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37849d8a-115b-44a5-bea8-539a92f53653-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.919939 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.919941 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37849d8a-115b-44a5-bea8-539a92f53653-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.920194 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.919957 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37849d8a-115b-44a5-bea8-539a92f53653-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.920194 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.920030 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37849d8a-115b-44a5-bea8-539a92f53653-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.920194 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.920068 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37849d8a-115b-44a5-bea8-539a92f53653-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.920194 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.920089 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37849d8a-115b-44a5-bea8-539a92f53653-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.920194 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.920114 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37849d8a-115b-44a5-bea8-539a92f53653-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.920194 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.920146 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pggkd\" (UniqueName: \"kubernetes.io/projected/37849d8a-115b-44a5-bea8-539a92f53653-kube-api-access-pggkd\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.920581 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.920244 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37849d8a-115b-44a5-bea8-539a92f53653-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.920581 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.920386 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37849d8a-115b-44a5-bea8-539a92f53653-workload-certs\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.920581 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.920498 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37849d8a-115b-44a5-bea8-539a92f53653-credential-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.920581 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.920516 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37849d8a-115b-44a5-bea8-539a92f53653-istio-data\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.920884 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.920717 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37849d8a-115b-44a5-bea8-539a92f53653-workload-socket\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.920884 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.920847 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37849d8a-115b-44a5-bea8-539a92f53653-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.922481 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.922452 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37849d8a-115b-44a5-bea8-539a92f53653-istio-envoy\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.922726 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.922681 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37849d8a-115b-44a5-bea8-539a92f53653-istio-podinfo\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.945602 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.945571 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37849d8a-115b-44a5-bea8-539a92f53653-istio-token\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:35.945741 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:35.945633 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pggkd\" (UniqueName: \"kubernetes.io/projected/37849d8a-115b-44a5-bea8-539a92f53653-kube-api-access-pggkd\") pod \"maas-default-gateway-openshift-default-c9ffffc7f-mjl62\" (UID: \"37849d8a-115b-44a5-bea8-539a92f53653\") " pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:36.027205 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.027166 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:36.341597 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.341514 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62"] Nov 03 17:39:36.355412 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:39:36.355386 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37849d8a_115b_44a5_bea8_539a92f53653.slice/crio-6060ed1103fb3efb797edd4c3f533555d78388e66ac4205c22ac18d3489f85c8 WatchSource:0}: Error finding container 6060ed1103fb3efb797edd4c3f533555d78388e66ac4205c22ac18d3489f85c8: Status 404 returned error can't find the container with id 6060ed1103fb3efb797edd4c3f533555d78388e66ac4205c22ac18d3489f85c8 Nov 03 17:39:36.357450 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.357417 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:39:36.357576 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.357496 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:39:36.357576 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.357542 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:39:36.429218 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.429192 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-54fd7b574c-db4h6"] Nov 03 17:39:36.432525 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.432505 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd7b574c-db4h6" Nov 03 17:39:36.448767 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.448739 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 17:39:36.452743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.452721 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sgj9n\"" Nov 03 17:39:36.452870 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.452768 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 17:39:36.452870 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.452768 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 17:39:36.464773 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.464745 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54fd7b574c-db4h6"] Nov 03 17:39:36.526269 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.526234 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61ba1e51-1854-4c3d-9918-d99dd92fbb18-metrics-certs\") pod \"maas-controller-54fd7b574c-db4h6\" (UID: \"61ba1e51-1854-4c3d-9918-d99dd92fbb18\") " pod="opendatahub/maas-controller-54fd7b574c-db4h6" Nov 03 17:39:36.526488 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.526322 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24zfr\" (UniqueName: \"kubernetes.io/projected/61ba1e51-1854-4c3d-9918-d99dd92fbb18-kube-api-access-24zfr\") pod \"maas-controller-54fd7b574c-db4h6\" (UID: \"61ba1e51-1854-4c3d-9918-d99dd92fbb18\") " pod="opendatahub/maas-controller-54fd7b574c-db4h6" Nov 03 17:39:36.526488 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.526342 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61ba1e51-1854-4c3d-9918-d99dd92fbb18-webhook-cert\") pod \"maas-controller-54fd7b574c-db4h6\" (UID: \"61ba1e51-1854-4c3d-9918-d99dd92fbb18\") " pod="opendatahub/maas-controller-54fd7b574c-db4h6" Nov 03 17:39:36.567161 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.567123 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb3dd1f-8e35-4520-b336-61ba4016aafa" path="/var/lib/kubelet/pods/cfb3dd1f-8e35-4520-b336-61ba4016aafa/volumes" Nov 03 17:39:36.627457 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.627368 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24zfr\" (UniqueName: \"kubernetes.io/projected/61ba1e51-1854-4c3d-9918-d99dd92fbb18-kube-api-access-24zfr\") pod \"maas-controller-54fd7b574c-db4h6\" (UID: \"61ba1e51-1854-4c3d-9918-d99dd92fbb18\") " pod="opendatahub/maas-controller-54fd7b574c-db4h6" Nov 03 17:39:36.627457 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.627411 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61ba1e51-1854-4c3d-9918-d99dd92fbb18-webhook-cert\") pod \"maas-controller-54fd7b574c-db4h6\" (UID: \"61ba1e51-1854-4c3d-9918-d99dd92fbb18\") " pod="opendatahub/maas-controller-54fd7b574c-db4h6" Nov 03 17:39:36.627743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.627489 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61ba1e51-1854-4c3d-9918-d99dd92fbb18-metrics-certs\") pod \"maas-controller-54fd7b574c-db4h6\" (UID: \"61ba1e51-1854-4c3d-9918-d99dd92fbb18\") " pod="opendatahub/maas-controller-54fd7b574c-db4h6" Nov 03 17:39:36.630282 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.630256 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61ba1e51-1854-4c3d-9918-d99dd92fbb18-metrics-certs\") pod \"maas-controller-54fd7b574c-db4h6\" (UID: \"61ba1e51-1854-4c3d-9918-d99dd92fbb18\") " pod="opendatahub/maas-controller-54fd7b574c-db4h6" Nov 03 17:39:36.630569 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.630540 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61ba1e51-1854-4c3d-9918-d99dd92fbb18-webhook-cert\") pod \"maas-controller-54fd7b574c-db4h6\" (UID: \"61ba1e51-1854-4c3d-9918-d99dd92fbb18\") " pod="opendatahub/maas-controller-54fd7b574c-db4h6" Nov 03 17:39:36.652371 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.652340 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24zfr\" (UniqueName: \"kubernetes.io/projected/61ba1e51-1854-4c3d-9918-d99dd92fbb18-kube-api-access-24zfr\") pod \"maas-controller-54fd7b574c-db4h6\" (UID: \"61ba1e51-1854-4c3d-9918-d99dd92fbb18\") " pod="opendatahub/maas-controller-54fd7b574c-db4h6" Nov 03 17:39:36.747797 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:36.747753 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-54fd7b574c-db4h6" Nov 03 17:39:37.070334 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:37.070305 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-54fd7b574c-db4h6"] Nov 03 17:39:37.071369 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:39:37.071338 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ba1e51_1854_4c3d_9918_d99dd92fbb18.slice/crio-6da92bca941668019b5a580cf261818554c282492b19d5f6e0dcdaf63f504035 WatchSource:0}: Error finding container 6da92bca941668019b5a580cf261818554c282492b19d5f6e0dcdaf63f504035: Status 404 returned error can't find the container with id 6da92bca941668019b5a580cf261818554c282492b19d5f6e0dcdaf63f504035 Nov 03 17:39:37.146765 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:37.146730 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd7b574c-db4h6" event={"ID":"61ba1e51-1854-4c3d-9918-d99dd92fbb18","Type":"ContainerStarted","Data":"6da92bca941668019b5a580cf261818554c282492b19d5f6e0dcdaf63f504035"} Nov 03 17:39:37.148387 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:37.148362 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" event={"ID":"37849d8a-115b-44a5-bea8-539a92f53653","Type":"ContainerStarted","Data":"5d8f821d35ab3aa72b34c415ce8cde0cb72230f443ae757e73491118975c9f20"} Nov 03 17:39:37.148525 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:37.148410 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" event={"ID":"37849d8a-115b-44a5-bea8-539a92f53653","Type":"ContainerStarted","Data":"6060ed1103fb3efb797edd4c3f533555d78388e66ac4205c22ac18d3489f85c8"} Nov 03 17:39:37.218724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:37.217864 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" podStartSLOduration=2.21784175 podStartE2EDuration="2.21784175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:39:37.196943559 +0000 UTC m=+5120.785943528" watchObservedRunningTime="2026-11-03 17:39:37.21784175 +0000 UTC m=+5120.806841719" Nov 03 17:39:38.027789 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:38.027748 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:38.029405 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:38.029369 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mjl62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Nov 03 17:39:38.029538 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:38.029437 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" podUID="37849d8a-115b-44a5-bea8-539a92f53653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Nov 03 17:39:38.155191 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:38.155156 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-54fd7b574c-db4h6" event={"ID":"61ba1e51-1854-4c3d-9918-d99dd92fbb18","Type":"ContainerStarted","Data":"909e0a75d43bcb922b2a85d21d4fe384dabba6362202972af115ba6798ad1dff"} Nov 03 17:39:38.155387 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:38.155240 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-54fd7b574c-db4h6" Nov 03 17:39:39.028244 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:39.028203 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mjl62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Nov 03 17:39:39.028773 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:39.028263 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" podUID="37849d8a-115b-44a5-bea8-539a92f53653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Nov 03 17:39:40.028339 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:40.028294 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mjl62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Nov 03 17:39:40.028901 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:40.028359 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" podUID="37849d8a-115b-44a5-bea8-539a92f53653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Nov 03 17:39:41.028305 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:41.028260 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mjl62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Nov 03 17:39:41.028484 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:41.028335 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" podUID="37849d8a-115b-44a5-bea8-539a92f53653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Nov 03 17:39:42.027978 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:42.027941 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mjl62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Nov 03 17:39:42.028169 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:42.027999 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" podUID="37849d8a-115b-44a5-bea8-539a92f53653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Nov 03 17:39:43.027954 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:43.027914 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mjl62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Nov 03 17:39:43.028366 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:43.027984 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" podUID="37849d8a-115b-44a5-bea8-539a92f53653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Nov 03 17:39:44.742824 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:44.742787 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:44.743465 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:44.743353 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:44.744053 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:44.744032 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" Nov 03 17:39:44.792068 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:44.791997 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-54fd7b574c-db4h6" podStartSLOduration=8.416488765 podStartE2EDuration="8.791977348s" totalImagesPullingTime="375.488583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:39:38.217158685 +0000 UTC m=+5121.806158655" watchObservedRunningTime="2026-11-03 17:39:44.791977348 +0000 UTC m=+5128.380977323" Nov 03 17:39:49.162831 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:49.162799 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-54fd7b574c-db4h6" Nov 03 17:39:57.689372 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:39:57.689329 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:40:00.269009 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:40:00.268963 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:40:00.792722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:40:00.790843 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mjl62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:40:00.792722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:40:00.790910 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" podUID="37849d8a-115b-44a5-bea8-539a92f53653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:40:13.565961 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:40:13.565923 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:40:18.906888 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:40:18.906840 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rp689 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:40:18.907820 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:40:18.907773 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" podUID="43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:40:19.016508 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:40:19.016469 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:40:25.525063 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:40:25.525022 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:40:29.799779 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:40:29.799681 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:40:30.128956 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:40:30.128864 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:41:00.128300 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:41:00.128264 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:41:05.268401 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:41:05.268365 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:41:13.579612 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:41:13.579573 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:41:30.129167 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:41:30.129123 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:41:49.661085 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:41:49.661045 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:42:00.129427 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:42:00.129341 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:42:08.558019 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:08.557980 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:42:15.035563 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:15.035524 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:42:24.198317 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.198275 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64"] Nov 03 17:42:24.202160 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.202134 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.218826 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.218803 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-4b1292-openshift-default-dockercfg-2gls8\"" Nov 03 17:42:24.237776 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.237749 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64"] Nov 03 17:42:24.340790 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.339191 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-envoy\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.340790 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.339268 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-workload-socket\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.340790 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.339312 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-credential-socket\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.340790 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.339372 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-data\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.340790 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.339403 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-podinfo\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.340790 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.339458 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8ll2\" (UniqueName: \"kubernetes.io/projected/6fcb2820-6a23-4b19-9d23-10c19e54b19c-kube-api-access-p8ll2\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.340790 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.339490 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istiod-ca-cert\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.340790 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.339538 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-token\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.340790 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.339568 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-workload-certs\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.441068 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.441028 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-workload-socket\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.441283 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.441083 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-credential-socket\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.441283 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.441144 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-data\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.441283 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.441166 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-podinfo\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.441283 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.441218 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8ll2\" (UniqueName: \"kubernetes.io/projected/6fcb2820-6a23-4b19-9d23-10c19e54b19c-kube-api-access-p8ll2\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.441283 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.441245 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istiod-ca-cert\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.442023 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.441287 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-token\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.442023 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.441319 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-workload-certs\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.442023 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.441401 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-envoy\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.442023 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.441583 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-workload-socket\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.442023 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.441628 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-data\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.442023 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.441871 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-workload-certs\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.442023 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.441905 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-credential-socket\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.442023 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.441958 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istiod-ca-cert\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.444335 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.444313 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-envoy\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.444561 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.444540 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-podinfo\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.466316 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.466253 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-token\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.466445 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.466376 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8ll2\" (UniqueName: \"kubernetes.io/projected/6fcb2820-6a23-4b19-9d23-10c19e54b19c-kube-api-access-p8ll2\") pod \"e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.516437 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.516393 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:24.841826 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.841792 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64"] Nov 03 17:42:24.841943 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:42:24.841851 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fcb2820_6a23_4b19_9d23_10c19e54b19c.slice/crio-3e0956b57472d1e39739d2957f7592ad2247b4b4965e901eb6329f19f40508ca WatchSource:0}: Error finding container 3e0956b57472d1e39739d2957f7592ad2247b4b4965e901eb6329f19f40508ca: Status 404 returned error can't find the container with id 3e0956b57472d1e39739d2957f7592ad2247b4b4965e901eb6329f19f40508ca Nov 03 17:42:24.844111 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.844075 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:42:24.844199 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.844149 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:42:24.844199 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:24.844181 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:42:25.250087 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:25.250052 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" event={"ID":"6fcb2820-6a23-4b19-9d23-10c19e54b19c","Type":"ContainerStarted","Data":"afd2c4a4f83c1e73e4e102903a29ec155e79a98f0a97e58e57066848d2b14621"} Nov 03 17:42:25.250485 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:25.250104 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" event={"ID":"6fcb2820-6a23-4b19-9d23-10c19e54b19c","Type":"ContainerStarted","Data":"3e0956b57472d1e39739d2957f7592ad2247b4b4965e901eb6329f19f40508ca"} Nov 03 17:42:25.325480 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:25.325432 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" podStartSLOduration=1.325419583 podStartE2EDuration="1.325419583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:25.310915292 +0000 UTC m=+5288.899915260" watchObservedRunningTime="2026-11-03 17:42:25.325419583 +0000 UTC m=+5288.914419551" Nov 03 17:42:25.517110 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:25.517018 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:25.522284 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:25.522260 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:26.254722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:26.254664 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:26.255592 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:26.255573 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:29.239495 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.239461 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5"] Nov 03 17:42:29.243198 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.243180 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" Nov 03 17:42:29.263076 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.263025 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-4b1292-serving-cert\"" Nov 03 17:42:29.279641 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.279612 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5"] Nov 03 17:42:29.390857 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.390824 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf52341-f786-4990-9968-80d7e3cca3cb-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5\" (UID: \"adf52341-f786-4990-9968-80d7e3cca3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" Nov 03 17:42:29.390857 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.390862 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf52341-f786-4990-9968-80d7e3cca3cb-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5\" (UID: \"adf52341-f786-4990-9968-80d7e3cca3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" Nov 03 17:42:29.391088 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.390900 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xtsp\" (UniqueName: \"kubernetes.io/projected/adf52341-f786-4990-9968-80d7e3cca3cb-kube-api-access-2xtsp\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5\" (UID: \"adf52341-f786-4990-9968-80d7e3cca3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" Nov 03 17:42:29.391088 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.391042 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adf52341-f786-4990-9968-80d7e3cca3cb-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5\" (UID: \"adf52341-f786-4990-9968-80d7e3cca3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" Nov 03 17:42:29.492589 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.492550 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adf52341-f786-4990-9968-80d7e3cca3cb-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5\" (UID: \"adf52341-f786-4990-9968-80d7e3cca3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" Nov 03 17:42:29.492864 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.492632 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf52341-f786-4990-9968-80d7e3cca3cb-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5\" (UID: \"adf52341-f786-4990-9968-80d7e3cca3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" Nov 03 17:42:29.492864 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.492654 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf52341-f786-4990-9968-80d7e3cca3cb-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5\" (UID: \"adf52341-f786-4990-9968-80d7e3cca3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" Nov 03 17:42:29.492864 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.492685 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtsp\" (UniqueName: \"kubernetes.io/projected/adf52341-f786-4990-9968-80d7e3cca3cb-kube-api-access-2xtsp\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5\" (UID: \"adf52341-f786-4990-9968-80d7e3cca3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" Nov 03 17:42:29.493341 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.493314 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf52341-f786-4990-9968-80d7e3cca3cb-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5\" (UID: \"adf52341-f786-4990-9968-80d7e3cca3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" Nov 03 17:42:29.495481 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.495451 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adf52341-f786-4990-9968-80d7e3cca3cb-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5\" (UID: \"adf52341-f786-4990-9968-80d7e3cca3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" Nov 03 17:42:29.495481 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.495470 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf52341-f786-4990-9968-80d7e3cca3cb-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5\" (UID: \"adf52341-f786-4990-9968-80d7e3cca3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" Nov 03 17:42:29.519935 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.519890 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xtsp\" (UniqueName: \"kubernetes.io/projected/adf52341-f786-4990-9968-80d7e3cca3cb-kube-api-access-2xtsp\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5\" (UID: \"adf52341-f786-4990-9968-80d7e3cca3cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" Nov 03 17:42:29.522674 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.522647 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb"] Nov 03 17:42:29.526343 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.526323 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" Nov 03 17:42:29.548282 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.548257 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-56341937-dockercfg-4fwz6\"" Nov 03 17:42:29.548933 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.548687 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-4b1292\"" Nov 03 17:42:29.550376 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.550353 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb"] Nov 03 17:42:29.557005 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.556985 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" Nov 03 17:42:29.586820 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.586733 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7"] Nov 03 17:42:29.592718 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.592675 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" Nov 03 17:42:29.596267 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.594076 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbl75\" (UniqueName: \"kubernetes.io/projected/48689cc2-7896-466c-99ef-9393f70275ee-kube-api-access-hbl75\") pod \"payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb\" (UID: \"48689cc2-7896-466c-99ef-9393f70275ee\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" Nov 03 17:42:29.596267 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.594176 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48689cc2-7896-466c-99ef-9393f70275ee-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb\" (UID: \"48689cc2-7896-466c-99ef-9393f70275ee\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" Nov 03 17:42:29.611976 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.611940 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7"] Nov 03 17:42:29.697261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.697223 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbl75\" (UniqueName: \"kubernetes.io/projected/48689cc2-7896-466c-99ef-9393f70275ee-kube-api-access-hbl75\") pod \"payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb\" (UID: \"48689cc2-7896-466c-99ef-9393f70275ee\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" Nov 03 17:42:29.697440 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.697338 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48689cc2-7896-466c-99ef-9393f70275ee-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb\" (UID: \"48689cc2-7896-466c-99ef-9393f70275ee\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" Nov 03 17:42:29.697440 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.697377 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pj49\" (UniqueName: \"kubernetes.io/projected/5040735c-b1cd-43b3-861f-0596546d2a90-kube-api-access-7pj49\") pod \"payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7\" (UID: \"5040735c-b1cd-43b3-861f-0596546d2a90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" Nov 03 17:42:29.697533 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.697440 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5040735c-b1cd-43b3-861f-0596546d2a90-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7\" (UID: \"5040735c-b1cd-43b3-861f-0596546d2a90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" Nov 03 17:42:29.698876 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.698847 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48689cc2-7896-466c-99ef-9393f70275ee-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb\" (UID: \"48689cc2-7896-466c-99ef-9393f70275ee\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" Nov 03 17:42:29.723791 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.723104 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbl75\" (UniqueName: \"kubernetes.io/projected/48689cc2-7896-466c-99ef-9393f70275ee-kube-api-access-hbl75\") pod \"payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb\" (UID: \"48689cc2-7896-466c-99ef-9393f70275ee\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" Nov 03 17:42:29.798136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.798053 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pj49\" (UniqueName: \"kubernetes.io/projected/5040735c-b1cd-43b3-861f-0596546d2a90-kube-api-access-7pj49\") pod \"payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7\" (UID: \"5040735c-b1cd-43b3-861f-0596546d2a90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" Nov 03 17:42:29.798136 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.798096 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5040735c-b1cd-43b3-861f-0596546d2a90-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7\" (UID: \"5040735c-b1cd-43b3-861f-0596546d2a90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" Nov 03 17:42:29.798820 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.798800 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5040735c-b1cd-43b3-861f-0596546d2a90-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7\" (UID: \"5040735c-b1cd-43b3-861f-0596546d2a90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" Nov 03 17:42:29.823476 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.823446 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pj49\" (UniqueName: \"kubernetes.io/projected/5040735c-b1cd-43b3-861f-0596546d2a90-kube-api-access-7pj49\") pod \"payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7\" (UID: \"5040735c-b1cd-43b3-861f-0596546d2a90\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" Nov 03 17:42:29.841039 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.841017 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" Nov 03 17:42:29.869406 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.869381 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5"] Nov 03 17:42:29.869843 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:42:29.869818 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf52341_f786_4990_9968_80d7e3cca3cb.slice/crio-b9af5b93763083d6cfd882eee1048aace5849f51e10f4bf34652c2a8b71487e1 WatchSource:0}: Error finding container b9af5b93763083d6cfd882eee1048aace5849f51e10f4bf34652c2a8b71487e1: Status 404 returned error can't find the container with id b9af5b93763083d6cfd882eee1048aace5849f51e10f4bf34652c2a8b71487e1 Nov 03 17:42:29.913083 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:29.913041 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" Nov 03 17:42:30.130043 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:42:30.129962 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:42:30.143569 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:30.143545 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb"] Nov 03 17:42:30.144097 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:42:30.144072 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48689cc2_7896_466c_99ef_9393f70275ee.slice/crio-3da0f9e185ba2912cf8d4631a6ef11e539a39a3fdbbd0b620172946f4407b918 WatchSource:0}: Error finding container 3da0f9e185ba2912cf8d4631a6ef11e539a39a3fdbbd0b620172946f4407b918: Status 404 returned error can't find the container with id 3da0f9e185ba2912cf8d4631a6ef11e539a39a3fdbbd0b620172946f4407b918 Nov 03 17:42:30.227940 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:30.227910 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7"] Nov 03 17:42:30.230130 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:42:30.230104 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5040735c_b1cd_43b3_861f_0596546d2a90.slice/crio-03e94aa16b77092423c6119010cd7372d6ac80287963fa635a45b77d0ac0b533 WatchSource:0}: Error finding container 03e94aa16b77092423c6119010cd7372d6ac80287963fa635a45b77d0ac0b533: Status 404 returned error can't find the container with id 03e94aa16b77092423c6119010cd7372d6ac80287963fa635a45b77d0ac0b533 Nov 03 17:42:30.282089 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:30.282061 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" event={"ID":"5040735c-b1cd-43b3-861f-0596546d2a90","Type":"ContainerStarted","Data":"03e94aa16b77092423c6119010cd7372d6ac80287963fa635a45b77d0ac0b533"} Nov 03 17:42:30.283265 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:30.283242 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" event={"ID":"adf52341-f786-4990-9968-80d7e3cca3cb","Type":"ContainerStarted","Data":"b9af5b93763083d6cfd882eee1048aace5849f51e10f4bf34652c2a8b71487e1"} Nov 03 17:42:30.284967 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:30.284939 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" event={"ID":"48689cc2-7896-466c-99ef-9393f70275ee","Type":"ContainerStarted","Data":"8413e0bf1fb1ce452604cb6fce7f10012f5fc8d2920e6dc3458e9afd8f55af57"} Nov 03 17:42:30.285094 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:30.284991 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" event={"ID":"48689cc2-7896-466c-99ef-9393f70275ee","Type":"ContainerStarted","Data":"3da0f9e185ba2912cf8d4631a6ef11e539a39a3fdbbd0b620172946f4407b918"} Nov 03 17:42:30.285282 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:30.285126 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" Nov 03 17:42:30.354758 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:30.342195 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" podStartSLOduration=1.342175035 podStartE2EDuration="1.342175035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:30.340776347 +0000 UTC m=+5293.929776319" watchObservedRunningTime="2026-11-03 17:42:30.342175035 +0000 UTC m=+5293.931175004" Nov 03 17:42:31.291024 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:31.290990 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" event={"ID":"5040735c-b1cd-43b3-861f-0596546d2a90","Type":"ContainerStarted","Data":"e9d413ef1d0df1048dd907995bf9f49c860c44f0747b31e85ce65732e8ed2495"} Nov 03 17:42:31.291498 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:31.291173 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" Nov 03 17:42:31.292754 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:31.292731 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" event={"ID":"adf52341-f786-4990-9968-80d7e3cca3cb","Type":"ContainerStarted","Data":"8b54ebcf78a1b30d422b1fd10e8d0a389e740739b75f036a1b8a43b6f34c989b"} Nov 03 17:42:31.292902 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:31.292881 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" Nov 03 17:42:31.336315 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:31.336264 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" podStartSLOduration=2.336250769 podStartE2EDuration="2.336250769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:31.335399248 +0000 UTC m=+5294.924399219" watchObservedRunningTime="2026-11-03 17:42:31.336250769 +0000 UTC m=+5294.925250734" Nov 03 17:42:31.382796 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:31.382741 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" podStartSLOduration=1.9511813340000002 podStartE2EDuration="2.382728315s" totalImagesPullingTime="431.546981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:31.380976948 +0000 UTC m=+5294.969976920" watchObservedRunningTime="2026-11-03 17:42:31.382728315 +0000 UTC m=+5294.971728284" Nov 03 17:42:32.892178 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:32.891995 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:42:37.327780 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:37.327746 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" Nov 03 17:42:41.296529 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:41.296490 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" Nov 03 17:42:42.296893 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:42.296838 4947 patch_prober.go:29] interesting pod/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:42:42.297460 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:42.296912 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" podUID="6fcb2820-6a23-4b19-9d23-10c19e54b19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:42:42.300489 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:42.300457 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" Nov 03 17:42:43.381795 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:43.381754 4947 generic.go:356] "Generic (PLEG): container finished" podID="6fcb2820-6a23-4b19-9d23-10c19e54b19c" containerID="afd2c4a4f83c1e73e4e102903a29ec155e79a98f0a97e58e57066848d2b14621" exitCode=0 Nov 03 17:42:43.382245 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:43.381826 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" event={"ID":"6fcb2820-6a23-4b19-9d23-10c19e54b19c","Type":"ContainerDied","Data":"afd2c4a4f83c1e73e4e102903a29ec155e79a98f0a97e58e57066848d2b14621"} Nov 03 17:42:43.382245 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:43.382215 4947 scope.go:122] "RemoveContainer" containerID="afd2c4a4f83c1e73e4e102903a29ec155e79a98f0a97e58e57066848d2b14621" Nov 03 17:42:43.384510 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:43.384470 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:42:43.384639 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:43.384570 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:42:43.384639 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:43.384626 4947 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Nov 03 17:42:44.389419 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:44.389374 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" event={"ID":"6fcb2820-6a23-4b19-9d23-10c19e54b19c","Type":"ContainerStarted","Data":"7b5122784ffc8bfbccd76ad91f10a12670fc0ee23247788fffd5efbf395ce8b4"} Nov 03 17:42:44.522728 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:44.520154 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:44.522963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:44.522821 4947 patch_prober.go:29] interesting pod/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Nov 03 17:42:44.522963 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:44.522892 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" podUID="6fcb2820-6a23-4b19-9d23-10c19e54b19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Nov 03 17:42:45.521724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:45.521090 4947 patch_prober.go:29] interesting pod/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Nov 03 17:42:45.521724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:45.521169 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" podUID="6fcb2820-6a23-4b19-9d23-10c19e54b19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Nov 03 17:42:46.519107 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:46.519068 4947 patch_prober.go:29] interesting pod/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Nov 03 17:42:46.519404 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:46.519373 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" podUID="6fcb2820-6a23-4b19-9d23-10c19e54b19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Nov 03 17:42:47.523167 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:47.523119 4947 patch_prober.go:29] interesting pod/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Nov 03 17:42:47.523680 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:47.523186 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" podUID="6fcb2820-6a23-4b19-9d23-10c19e54b19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Nov 03 17:42:47.949834 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:47.949753 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:42:48.522728 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:48.518981 4947 patch_prober.go:29] interesting pod/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Nov 03 17:42:48.522728 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:48.519043 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" podUID="6fcb2820-6a23-4b19-9d23-10c19e54b19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Nov 03 17:42:49.517834 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:49.517779 4947 patch_prober.go:29] interesting pod/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" start-of-body= Nov 03 17:42:49.518261 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:49.517864 4947 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" podUID="6fcb2820-6a23-4b19-9d23-10c19e54b19c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.33:15021/healthz/ready\": dial tcp 10.132.1.33:15021: connect: connection refused" Nov 03 17:42:50.520681 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:50.520649 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:50.521100 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:50.520954 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:42:50.521595 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:42:50.521578 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:43:00.129341 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:43:00.129300 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:43:03.056329 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:03.056289 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:43:26.849617 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:26.849583 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:43:30.130185 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:43:30.130146 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:43:32.216291 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:32.216251 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb"] Nov 03 17:43:32.249763 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:32.249732 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7"] Nov 03 17:43:33.326012 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:33.325973 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5"] Nov 03 17:43:33.326567 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:33.326214 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" podUID="adf52341-f786-4990-9968-80d7e3cca3cb" containerName="maas-api" containerID="cri-o://8b54ebcf78a1b30d422b1fd10e8d0a389e740739b75f036a1b8a43b6f34c989b" gracePeriod=30 Nov 03 17:43:33.657776 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:33.657749 4947 generic.go:356] "Generic (PLEG): container finished" podID="adf52341-f786-4990-9968-80d7e3cca3cb" containerID="8b54ebcf78a1b30d422b1fd10e8d0a389e740739b75f036a1b8a43b6f34c989b" exitCode=0 Nov 03 17:43:33.657860 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:33.657831 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" event={"ID":"adf52341-f786-4990-9968-80d7e3cca3cb","Type":"ContainerDied","Data":"8b54ebcf78a1b30d422b1fd10e8d0a389e740739b75f036a1b8a43b6f34c989b"} Nov 03 17:43:33.838556 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:33.838526 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" Nov 03 17:43:33.933571 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:33.933484 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adf52341-f786-4990-9968-80d7e3cca3cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/adf52341-f786-4990-9968-80d7e3cca3cb-metrics-certs\") pod \"adf52341-f786-4990-9968-80d7e3cca3cb\" (UID: \"adf52341-f786-4990-9968-80d7e3cca3cb\") " Nov 03 17:43:33.933571 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:33.933542 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adf52341-f786-4990-9968-80d7e3cca3cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf52341-f786-4990-9968-80d7e3cca3cb-openshift-service-ca\") pod \"adf52341-f786-4990-9968-80d7e3cca3cb\" (UID: \"adf52341-f786-4990-9968-80d7e3cca3cb\") " Nov 03 17:43:33.933896 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:33.933572 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adf52341-f786-4990-9968-80d7e3cca3cb-kube-api-access-2xtsp\" (UniqueName: \"kubernetes.io/projected/adf52341-f786-4990-9968-80d7e3cca3cb-kube-api-access-2xtsp\") pod \"adf52341-f786-4990-9968-80d7e3cca3cb\" (UID: \"adf52341-f786-4990-9968-80d7e3cca3cb\") " Nov 03 17:43:33.933896 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:33.933729 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adf52341-f786-4990-9968-80d7e3cca3cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf52341-f786-4990-9968-80d7e3cca3cb-maas-api-tls\") pod \"adf52341-f786-4990-9968-80d7e3cca3cb\" (UID: \"adf52341-f786-4990-9968-80d7e3cca3cb\") " Nov 03 17:43:33.933999 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:33.933936 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adf52341-f786-4990-9968-80d7e3cca3cb-openshift-service-ca" pod "adf52341-f786-4990-9968-80d7e3cca3cb" (UID: "adf52341-f786-4990-9968-80d7e3cca3cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:43:33.934120 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:33.934102 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/adf52341-f786-4990-9968-80d7e3cca3cb-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:43:33.935918 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:33.935888 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adf52341-f786-4990-9968-80d7e3cca3cb-maas-api-tls" pod "adf52341-f786-4990-9968-80d7e3cca3cb" (UID: "adf52341-f786-4990-9968-80d7e3cca3cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:43:33.936040 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:33.935969 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adf52341-f786-4990-9968-80d7e3cca3cb-metrics-certs" pod "adf52341-f786-4990-9968-80d7e3cca3cb" (UID: "adf52341-f786-4990-9968-80d7e3cca3cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:43:33.936040 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:33.936012 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adf52341-f786-4990-9968-80d7e3cca3cb-kube-api-access-2xtsp" pod "adf52341-f786-4990-9968-80d7e3cca3cb" (UID: "adf52341-f786-4990-9968-80d7e3cca3cb"). InnerVolumeSpecName "kube-api-access-2xtsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:43:34.034579 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:34.034543 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/adf52341-f786-4990-9968-80d7e3cca3cb-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:43:34.034579 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:34.034575 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adf52341-f786-4990-9968-80d7e3cca3cb-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:43:34.034579 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:34.034585 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xtsp\" (UniqueName: \"kubernetes.io/projected/adf52341-f786-4990-9968-80d7e3cca3cb-kube-api-access-2xtsp\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:43:34.663870 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:34.663840 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" Nov 03 17:43:34.664326 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:34.663846 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5" event={"ID":"adf52341-f786-4990-9968-80d7e3cca3cb","Type":"ContainerDied","Data":"b9af5b93763083d6cfd882eee1048aace5849f51e10f4bf34652c2a8b71487e1"} Nov 03 17:43:34.664326 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:34.663974 4947 scope.go:122] "RemoveContainer" containerID="8b54ebcf78a1b30d422b1fd10e8d0a389e740739b75f036a1b8a43b6f34c989b" Nov 03 17:43:34.725617 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:34.725592 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5"] Nov 03 17:43:34.738991 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:34.738969 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-dgjm5"] Nov 03 17:43:35.882386 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:35.882334 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:43:36.563686 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:36.563651 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adf52341-f786-4990-9968-80d7e3cca3cb" path="/var/lib/kubelet/pods/adf52341-f786-4990-9968-80d7e3cca3cb/volumes" Nov 03 17:43:37.217472 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:37.217399 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" podUID="48689cc2-7896-466c-99ef-9393f70275ee" containerName="payload-processing" containerID="cri-o://8413e0bf1fb1ce452604cb6fce7f10012f5fc8d2920e6dc3458e9afd8f55af57" gracePeriod=25 Nov 03 17:43:37.250973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:37.250934 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" podUID="5040735c-b1cd-43b3-861f-0596546d2a90" containerName="payload-pre-processing" containerID="cri-o://e9d413ef1d0df1048dd907995bf9f49c860c44f0747b31e85ce65732e8ed2495" gracePeriod=25 Nov 03 17:43:41.294800 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:41.294759 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" podUID="48689cc2-7896-466c-99ef-9393f70275ee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:43:42.298602 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:42.298561 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" podUID="5040735c-b1cd-43b3-861f-0596546d2a90" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:43:51.294858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:51.294818 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" podUID="48689cc2-7896-466c-99ef-9393f70275ee" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:43:52.298496 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:52.298460 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" podUID="5040735c-b1cd-43b3-861f-0596546d2a90" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 17:43:57.509048 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.509022 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" Nov 03 17:43:57.534860 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.534840 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" Nov 03 17:43:57.662448 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.662416 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48689cc2-7896-466c-99ef-9393f70275ee-kube-api-access-hbl75\" (UniqueName: \"kubernetes.io/projected/48689cc2-7896-466c-99ef-9393f70275ee-kube-api-access-hbl75\") pod \"48689cc2-7896-466c-99ef-9393f70275ee\" (UID: \"48689cc2-7896-466c-99ef-9393f70275ee\") " Nov 03 17:43:57.662620 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.662551 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5040735c-b1cd-43b3-861f-0596546d2a90-kube-api-access-7pj49\" (UniqueName: \"kubernetes.io/projected/5040735c-b1cd-43b3-861f-0596546d2a90-kube-api-access-7pj49\") pod \"5040735c-b1cd-43b3-861f-0596546d2a90\" (UID: \"5040735c-b1cd-43b3-861f-0596546d2a90\") " Nov 03 17:43:57.662620 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.662571 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5040735c-b1cd-43b3-861f-0596546d2a90-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5040735c-b1cd-43b3-861f-0596546d2a90-plugins-config-volume\") pod \"5040735c-b1cd-43b3-861f-0596546d2a90\" (UID: \"5040735c-b1cd-43b3-861f-0596546d2a90\") " Nov 03 17:43:57.662620 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.662601 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48689cc2-7896-466c-99ef-9393f70275ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48689cc2-7896-466c-99ef-9393f70275ee-plugins-config-volume\") pod \"48689cc2-7896-466c-99ef-9393f70275ee\" (UID: \"48689cc2-7896-466c-99ef-9393f70275ee\") " Nov 03 17:43:57.663045 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.663021 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5040735c-b1cd-43b3-861f-0596546d2a90-plugins-config-volume" pod "5040735c-b1cd-43b3-861f-0596546d2a90" (UID: "5040735c-b1cd-43b3-861f-0596546d2a90"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:43:57.663126 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.663039 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48689cc2-7896-466c-99ef-9393f70275ee-plugins-config-volume" pod "48689cc2-7896-466c-99ef-9393f70275ee" (UID: "48689cc2-7896-466c-99ef-9393f70275ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:43:57.664876 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.664848 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48689cc2-7896-466c-99ef-9393f70275ee-kube-api-access-hbl75" pod "48689cc2-7896-466c-99ef-9393f70275ee" (UID: "48689cc2-7896-466c-99ef-9393f70275ee"). InnerVolumeSpecName "kube-api-access-hbl75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:43:57.665002 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.664923 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5040735c-b1cd-43b3-861f-0596546d2a90-kube-api-access-7pj49" pod "5040735c-b1cd-43b3-861f-0596546d2a90" (UID: "5040735c-b1cd-43b3-861f-0596546d2a90"). InnerVolumeSpecName "kube-api-access-7pj49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:43:57.764298 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.764258 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbl75\" (UniqueName: \"kubernetes.io/projected/48689cc2-7896-466c-99ef-9393f70275ee-kube-api-access-hbl75\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:43:57.764298 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.764297 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pj49\" (UniqueName: \"kubernetes.io/projected/5040735c-b1cd-43b3-861f-0596546d2a90-kube-api-access-7pj49\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:43:57.764530 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.764307 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5040735c-b1cd-43b3-861f-0596546d2a90-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:43:57.764530 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.764318 4947 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48689cc2-7896-466c-99ef-9393f70275ee-plugins-config-volume\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:43:57.776295 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.776268 4947 generic.go:356] "Generic (PLEG): container finished" podID="48689cc2-7896-466c-99ef-9393f70275ee" containerID="8413e0bf1fb1ce452604cb6fce7f10012f5fc8d2920e6dc3458e9afd8f55af57" exitCode=0 Nov 03 17:43:57.776448 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.776319 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" Nov 03 17:43:57.776448 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.776363 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" event={"ID":"48689cc2-7896-466c-99ef-9393f70275ee","Type":"ContainerDied","Data":"8413e0bf1fb1ce452604cb6fce7f10012f5fc8d2920e6dc3458e9afd8f55af57"} Nov 03 17:43:57.776448 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.776410 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb" event={"ID":"48689cc2-7896-466c-99ef-9393f70275ee","Type":"ContainerDied","Data":"3da0f9e185ba2912cf8d4631a6ef11e539a39a3fdbbd0b620172946f4407b918"} Nov 03 17:43:57.776448 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.776431 4947 scope.go:122] "RemoveContainer" containerID="8413e0bf1fb1ce452604cb6fce7f10012f5fc8d2920e6dc3458e9afd8f55af57" Nov 03 17:43:57.777788 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.777763 4947 generic.go:356] "Generic (PLEG): container finished" podID="5040735c-b1cd-43b3-861f-0596546d2a90" containerID="e9d413ef1d0df1048dd907995bf9f49c860c44f0747b31e85ce65732e8ed2495" exitCode=0 Nov 03 17:43:57.777879 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.777802 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" Nov 03 17:43:57.777879 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.777834 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" event={"ID":"5040735c-b1cd-43b3-861f-0596546d2a90","Type":"ContainerDied","Data":"e9d413ef1d0df1048dd907995bf9f49c860c44f0747b31e85ce65732e8ed2495"} Nov 03 17:43:57.777879 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.777874 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7" event={"ID":"5040735c-b1cd-43b3-861f-0596546d2a90","Type":"ContainerDied","Data":"03e94aa16b77092423c6119010cd7372d6ac80287963fa635a45b77d0ac0b533"} Nov 03 17:43:57.787230 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.787209 4947 scope.go:122] "RemoveContainer" containerID="8413e0bf1fb1ce452604cb6fce7f10012f5fc8d2920e6dc3458e9afd8f55af57" Nov 03 17:43:57.787572 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:43:57.787541 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8413e0bf1fb1ce452604cb6fce7f10012f5fc8d2920e6dc3458e9afd8f55af57\": container with ID starting with 8413e0bf1fb1ce452604cb6fce7f10012f5fc8d2920e6dc3458e9afd8f55af57 not found: ID does not exist" containerID="8413e0bf1fb1ce452604cb6fce7f10012f5fc8d2920e6dc3458e9afd8f55af57" Nov 03 17:43:57.787652 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.787587 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8413e0bf1fb1ce452604cb6fce7f10012f5fc8d2920e6dc3458e9afd8f55af57"} err="failed to get container status \"8413e0bf1fb1ce452604cb6fce7f10012f5fc8d2920e6dc3458e9afd8f55af57\": rpc error: code = NotFound desc = could not find container \"8413e0bf1fb1ce452604cb6fce7f10012f5fc8d2920e6dc3458e9afd8f55af57\": container with ID starting with 8413e0bf1fb1ce452604cb6fce7f10012f5fc8d2920e6dc3458e9afd8f55af57 not found: ID does not exist" Nov 03 17:43:57.787652 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.787613 4947 scope.go:122] "RemoveContainer" containerID="e9d413ef1d0df1048dd907995bf9f49c860c44f0747b31e85ce65732e8ed2495" Nov 03 17:43:57.795666 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.795649 4947 scope.go:122] "RemoveContainer" containerID="e9d413ef1d0df1048dd907995bf9f49c860c44f0747b31e85ce65732e8ed2495" Nov 03 17:43:57.795963 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:43:57.795944 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9d413ef1d0df1048dd907995bf9f49c860c44f0747b31e85ce65732e8ed2495\": container with ID starting with e9d413ef1d0df1048dd907995bf9f49c860c44f0747b31e85ce65732e8ed2495 not found: ID does not exist" containerID="e9d413ef1d0df1048dd907995bf9f49c860c44f0747b31e85ce65732e8ed2495" Nov 03 17:43:57.796034 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.795973 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9d413ef1d0df1048dd907995bf9f49c860c44f0747b31e85ce65732e8ed2495"} err="failed to get container status \"e9d413ef1d0df1048dd907995bf9f49c860c44f0747b31e85ce65732e8ed2495\": rpc error: code = NotFound desc = could not find container \"e9d413ef1d0df1048dd907995bf9f49c860c44f0747b31e85ce65732e8ed2495\": container with ID starting with e9d413ef1d0df1048dd907995bf9f49c860c44f0747b31e85ce65732e8ed2495 not found: ID does not exist" Nov 03 17:43:57.831523 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.831498 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb"] Nov 03 17:43:57.835132 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.835111 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4b1292-67dbc6d85txktb"] Nov 03 17:43:57.870862 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.870838 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7"] Nov 03 17:43:57.874897 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:57.874878 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-4b1292-675d7stmz7"] Nov 03 17:43:58.565098 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:58.565065 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48689cc2-7896-466c-99ef-9393f70275ee" path="/var/lib/kubelet/pods/48689cc2-7896-466c-99ef-9393f70275ee/volumes" Nov 03 17:43:58.565507 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:43:58.565440 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5040735c-b1cd-43b3-861f-0596546d2a90" path="/var/lib/kubelet/pods/5040735c-b1cd-43b3-861f-0596546d2a90/volumes" Nov 03 17:44:00.129540 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:44:00.129506 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:44:05.519256 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.519221 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd"] Nov 03 17:44:05.519837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.519622 4947 cpu_manager.go:414] "containerMap: removing container" podUID="5040735c-b1cd-43b3-861f-0596546d2a90" containerName="payload-pre-processing" Nov 03 17:44:05.519837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.519633 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5040735c-b1cd-43b3-861f-0596546d2a90" containerName="payload-pre-processing" Nov 03 17:44:05.519837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.519647 4947 cpu_manager.go:414] "containerMap: removing container" podUID="48689cc2-7896-466c-99ef-9393f70275ee" containerName="payload-processing" Nov 03 17:44:05.519837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.519652 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48689cc2-7896-466c-99ef-9393f70275ee" containerName="payload-processing" Nov 03 17:44:05.519837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.519660 4947 cpu_manager.go:414] "containerMap: removing container" podUID="adf52341-f786-4990-9968-80d7e3cca3cb" containerName="maas-api" Nov 03 17:44:05.519837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.519664 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adf52341-f786-4990-9968-80d7e3cca3cb" containerName="maas-api" Nov 03 17:44:05.519837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.519769 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="5040735c-b1cd-43b3-861f-0596546d2a90" containerName="payload-pre-processing" Nov 03 17:44:05.519837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.519778 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="48689cc2-7896-466c-99ef-9393f70275ee" containerName="payload-processing" Nov 03 17:44:05.519837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.519784 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="adf52341-f786-4990-9968-80d7e3cca3cb" containerName="maas-api" Nov 03 17:44:05.523231 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.523211 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" Nov 03 17:44:05.555153 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.555113 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-4b1292-serving-cert\"" Nov 03 17:44:05.557172 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.557141 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd"] Nov 03 17:44:05.636069 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.636036 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6111883c-34de-4de0-aa85-e4849e24e67a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd\" (UID: \"6111883c-34de-4de0-aa85-e4849e24e67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" Nov 03 17:44:05.636260 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.636142 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6111883c-34de-4de0-aa85-e4849e24e67a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd\" (UID: \"6111883c-34de-4de0-aa85-e4849e24e67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" Nov 03 17:44:05.636260 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.636176 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nclrk\" (UniqueName: \"kubernetes.io/projected/6111883c-34de-4de0-aa85-e4849e24e67a-kube-api-access-nclrk\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd\" (UID: \"6111883c-34de-4de0-aa85-e4849e24e67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" Nov 03 17:44:05.636260 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.636211 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6111883c-34de-4de0-aa85-e4849e24e67a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd\" (UID: \"6111883c-34de-4de0-aa85-e4849e24e67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" Nov 03 17:44:05.736884 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.736850 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6111883c-34de-4de0-aa85-e4849e24e67a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd\" (UID: \"6111883c-34de-4de0-aa85-e4849e24e67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" Nov 03 17:44:05.737076 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.736913 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6111883c-34de-4de0-aa85-e4849e24e67a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd\" (UID: \"6111883c-34de-4de0-aa85-e4849e24e67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" Nov 03 17:44:05.737076 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.736942 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nclrk\" (UniqueName: \"kubernetes.io/projected/6111883c-34de-4de0-aa85-e4849e24e67a-kube-api-access-nclrk\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd\" (UID: \"6111883c-34de-4de0-aa85-e4849e24e67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" Nov 03 17:44:05.737076 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.736983 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6111883c-34de-4de0-aa85-e4849e24e67a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd\" (UID: \"6111883c-34de-4de0-aa85-e4849e24e67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" Nov 03 17:44:05.737833 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.737807 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6111883c-34de-4de0-aa85-e4849e24e67a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd\" (UID: \"6111883c-34de-4de0-aa85-e4849e24e67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" Nov 03 17:44:05.739609 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.739586 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6111883c-34de-4de0-aa85-e4849e24e67a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd\" (UID: \"6111883c-34de-4de0-aa85-e4849e24e67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" Nov 03 17:44:05.739769 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.739729 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6111883c-34de-4de0-aa85-e4849e24e67a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd\" (UID: \"6111883c-34de-4de0-aa85-e4849e24e67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" Nov 03 17:44:05.765751 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.765721 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nclrk\" (UniqueName: \"kubernetes.io/projected/6111883c-34de-4de0-aa85-e4849e24e67a-kube-api-access-nclrk\") pod \"maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd\" (UID: \"6111883c-34de-4de0-aa85-e4849e24e67a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" Nov 03 17:44:05.836010 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:05.835924 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" Nov 03 17:44:06.140325 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:06.140297 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd"] Nov 03 17:44:06.141159 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:44:06.141131 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6111883c_34de_4de0_aa85_e4849e24e67a.slice/crio-fac9cb2da5b936d209bc52164c32aff3b3e8a110dad5b535278438a788894670 WatchSource:0}: Error finding container fac9cb2da5b936d209bc52164c32aff3b3e8a110dad5b535278438a788894670: Status 404 returned error can't find the container with id fac9cb2da5b936d209bc52164c32aff3b3e8a110dad5b535278438a788894670 Nov 03 17:44:06.142416 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:06.142395 4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:44:06.821259 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:06.821224 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" event={"ID":"6111883c-34de-4de0-aa85-e4849e24e67a","Type":"ContainerStarted","Data":"9a7862f41493cbb622fb059be11c9507860a9879f59f794688408374957fa24c"} Nov 03 17:44:06.821652 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:06.821305 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" event={"ID":"6111883c-34de-4de0-aa85-e4849e24e67a","Type":"ContainerStarted","Data":"fac9cb2da5b936d209bc52164c32aff3b3e8a110dad5b535278438a788894670"} Nov 03 17:44:06.821652 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:06.821329 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" Nov 03 17:44:06.868426 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:06.868368 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" podStartSLOduration=1.526637121 podStartE2EDuration="1.868354258s" totalImagesPullingTime="341.717137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:06.866513125 +0000 UTC m=+5390.455513094" watchObservedRunningTime="2026-11-03 17:44:06.868354258 +0000 UTC m=+5390.457354253" Nov 03 17:44:09.655065 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:09.655034 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:44:12.833096 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:12.833067 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" Nov 03 17:44:17.418458 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:17.418428 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:44:17.425867 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:17.425836 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:44:26.643060 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:26.643008 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd"] Nov 03 17:44:26.643921 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:26.643878 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" podUID="6111883c-34de-4de0-aa85-e4849e24e67a" containerName="maas-api" containerID="cri-o://9a7862f41493cbb622fb059be11c9507860a9879f59f794688408374957fa24c" gracePeriod=30 Nov 03 17:44:26.922973 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:26.922891 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" Nov 03 17:44:26.929333 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:26.929302 4947 generic.go:356] "Generic (PLEG): container finished" podID="6111883c-34de-4de0-aa85-e4849e24e67a" containerID="9a7862f41493cbb622fb059be11c9507860a9879f59f794688408374957fa24c" exitCode=0 Nov 03 17:44:26.929516 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:26.929349 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" event={"ID":"6111883c-34de-4de0-aa85-e4849e24e67a","Type":"ContainerDied","Data":"9a7862f41493cbb622fb059be11c9507860a9879f59f794688408374957fa24c"} Nov 03 17:44:26.929516 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:26.929367 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" Nov 03 17:44:26.929516 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:26.929402 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd" event={"ID":"6111883c-34de-4de0-aa85-e4849e24e67a","Type":"ContainerDied","Data":"fac9cb2da5b936d209bc52164c32aff3b3e8a110dad5b535278438a788894670"} Nov 03 17:44:26.929516 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:26.929431 4947 scope.go:122] "RemoveContainer" containerID="9a7862f41493cbb622fb059be11c9507860a9879f59f794688408374957fa24c" Nov 03 17:44:26.940411 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:26.940391 4947 scope.go:122] "RemoveContainer" containerID="9a7862f41493cbb622fb059be11c9507860a9879f59f794688408374957fa24c" Nov 03 17:44:26.940747 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:44:26.940729 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a7862f41493cbb622fb059be11c9507860a9879f59f794688408374957fa24c\": container with ID starting with 9a7862f41493cbb622fb059be11c9507860a9879f59f794688408374957fa24c not found: ID does not exist" containerID="9a7862f41493cbb622fb059be11c9507860a9879f59f794688408374957fa24c" Nov 03 17:44:26.940823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:26.940757 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a7862f41493cbb622fb059be11c9507860a9879f59f794688408374957fa24c"} err="failed to get container status \"9a7862f41493cbb622fb059be11c9507860a9879f59f794688408374957fa24c\": rpc error: code = NotFound desc = could not find container \"9a7862f41493cbb622fb059be11c9507860a9879f59f794688408374957fa24c\": container with ID starting with 9a7862f41493cbb622fb059be11c9507860a9879f59f794688408374957fa24c not found: ID does not exist" Nov 03 17:44:27.055873 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:27.055838 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6111883c-34de-4de0-aa85-e4849e24e67a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6111883c-34de-4de0-aa85-e4849e24e67a-maas-api-tls\") pod \"6111883c-34de-4de0-aa85-e4849e24e67a\" (UID: \"6111883c-34de-4de0-aa85-e4849e24e67a\") " Nov 03 17:44:27.056078 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:27.055915 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6111883c-34de-4de0-aa85-e4849e24e67a-kube-api-access-nclrk\" (UniqueName: \"kubernetes.io/projected/6111883c-34de-4de0-aa85-e4849e24e67a-kube-api-access-nclrk\") pod \"6111883c-34de-4de0-aa85-e4849e24e67a\" (UID: \"6111883c-34de-4de0-aa85-e4849e24e67a\") " Nov 03 17:44:27.056078 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:27.055953 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6111883c-34de-4de0-aa85-e4849e24e67a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6111883c-34de-4de0-aa85-e4849e24e67a-metrics-certs\") pod \"6111883c-34de-4de0-aa85-e4849e24e67a\" (UID: \"6111883c-34de-4de0-aa85-e4849e24e67a\") " Nov 03 17:44:27.056215 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:27.056118 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6111883c-34de-4de0-aa85-e4849e24e67a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6111883c-34de-4de0-aa85-e4849e24e67a-openshift-service-ca\") pod \"6111883c-34de-4de0-aa85-e4849e24e67a\" (UID: \"6111883c-34de-4de0-aa85-e4849e24e67a\") " Nov 03 17:44:27.056548 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:27.056513 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6111883c-34de-4de0-aa85-e4849e24e67a-openshift-service-ca" pod "6111883c-34de-4de0-aa85-e4849e24e67a" (UID: "6111883c-34de-4de0-aa85-e4849e24e67a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:44:27.058208 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:27.058185 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6111883c-34de-4de0-aa85-e4849e24e67a-maas-api-tls" pod "6111883c-34de-4de0-aa85-e4849e24e67a" (UID: "6111883c-34de-4de0-aa85-e4849e24e67a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:44:27.058728 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:27.058692 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6111883c-34de-4de0-aa85-e4849e24e67a-metrics-certs" pod "6111883c-34de-4de0-aa85-e4849e24e67a" (UID: "6111883c-34de-4de0-aa85-e4849e24e67a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:44:27.058829 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:27.058740 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6111883c-34de-4de0-aa85-e4849e24e67a-kube-api-access-nclrk" pod "6111883c-34de-4de0-aa85-e4849e24e67a" (UID: "6111883c-34de-4de0-aa85-e4849e24e67a"). InnerVolumeSpecName "kube-api-access-nclrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:44:27.157403 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:27.157348 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6111883c-34de-4de0-aa85-e4849e24e67a-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:44:27.157403 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:27.157385 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nclrk\" (UniqueName: \"kubernetes.io/projected/6111883c-34de-4de0-aa85-e4849e24e67a-kube-api-access-nclrk\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:44:27.157403 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:27.157400 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6111883c-34de-4de0-aa85-e4849e24e67a-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:44:27.157403 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:27.157411 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6111883c-34de-4de0-aa85-e4849e24e67a-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:44:27.277812 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:27.277781 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd"] Nov 03 17:44:27.291006 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:27.290978 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-4b1292-6d98b877d6-6m9kd"] Nov 03 17:44:27.806805 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:27.806751 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64"] Nov 03 17:44:27.807319 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:27.806991 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" podUID="6fcb2820-6a23-4b19-9d23-10c19e54b19c" containerName="istio-proxy" containerID="cri-o://7b5122784ffc8bfbccd76ad91f10a12670fc0ee23247788fffd5efbf395ce8b4" gracePeriod=30 Nov 03 17:44:28.564799 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:28.564759 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6111883c-34de-4de0-aa85-e4849e24e67a" path="/var/lib/kubelet/pods/6111883c-34de-4de0-aa85-e4849e24e67a/volumes" Nov 03 17:44:29.742721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:29.742667 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:44:30.129984 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:44:30.129902 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:44:32.965968 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:32.965930 4947 generic.go:356] "Generic (PLEG): container finished" podID="6fcb2820-6a23-4b19-9d23-10c19e54b19c" containerID="7b5122784ffc8bfbccd76ad91f10a12670fc0ee23247788fffd5efbf395ce8b4" exitCode=0 Nov 03 17:44:32.966409 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:32.965999 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" event={"ID":"6fcb2820-6a23-4b19-9d23-10c19e54b19c","Type":"ContainerDied","Data":"7b5122784ffc8bfbccd76ad91f10a12670fc0ee23247788fffd5efbf395ce8b4"} Nov 03 17:44:32.966409 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:32.966064 4947 scope.go:122] "RemoveContainer" containerID="afd2c4a4f83c1e73e4e102903a29ec155e79a98f0a97e58e57066848d2b14621" Nov 03 17:44:33.070745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.070677 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:44:33.225797 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.225767 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fcb2820-6a23-4b19-9d23-10c19e54b19c-kube-api-access-p8ll2\" (UniqueName: \"kubernetes.io/projected/6fcb2820-6a23-4b19-9d23-10c19e54b19c-kube-api-access-p8ll2\") pod \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " Nov 03 17:44:33.225976 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.225803 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istiod-ca-cert\") pod \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " Nov 03 17:44:33.225976 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.225821 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-token\" (UniqueName: \"kubernetes.io/projected/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-token\") pod \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " Nov 03 17:44:33.226081 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.226007 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-envoy\") pod \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " Nov 03 17:44:33.226081 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.226048 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-data\") pod \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " Nov 03 17:44:33.226197 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.226084 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-workload-socket\") pod \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " Nov 03 17:44:33.226197 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.226159 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-podinfo\") pod \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " Nov 03 17:44:33.226304 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.226215 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-workload-certs\") pod \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " Nov 03 17:44:33.226304 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.226211 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istiod-ca-cert" pod "6fcb2820-6a23-4b19-9d23-10c19e54b19c" (UID: "6fcb2820-6a23-4b19-9d23-10c19e54b19c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:44:33.226304 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.226266 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-credential-socket\") pod \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\" (UID: \"6fcb2820-6a23-4b19-9d23-10c19e54b19c\") " Nov 03 17:44:33.226487 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.226343 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-data" pod "6fcb2820-6a23-4b19-9d23-10c19e54b19c" (UID: "6fcb2820-6a23-4b19-9d23-10c19e54b19c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:44:33.226556 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.226473 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-workload-socket" pod "6fcb2820-6a23-4b19-9d23-10c19e54b19c" (UID: "6fcb2820-6a23-4b19-9d23-10c19e54b19c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:44:33.226714 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.226629 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-workload-certs" pod "6fcb2820-6a23-4b19-9d23-10c19e54b19c" (UID: "6fcb2820-6a23-4b19-9d23-10c19e54b19c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:44:33.226714 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.226643 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-credential-socket" pod "6fcb2820-6a23-4b19-9d23-10c19e54b19c" (UID: "6fcb2820-6a23-4b19-9d23-10c19e54b19c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:44:33.226914 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.226806 4947 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-workload-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:44:33.226914 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.226826 4947 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-credential-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:44:33.226914 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.226840 4947 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istiod-ca-cert\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:44:33.226914 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.226853 4947 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-data\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:44:33.226914 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.226865 4947 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-workload-socket\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:44:33.228387 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.228359 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fcb2820-6a23-4b19-9d23-10c19e54b19c-kube-api-access-p8ll2" pod "6fcb2820-6a23-4b19-9d23-10c19e54b19c" (UID: "6fcb2820-6a23-4b19-9d23-10c19e54b19c"). InnerVolumeSpecName "kube-api-access-p8ll2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:44:33.228474 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.228393 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-token" pod "6fcb2820-6a23-4b19-9d23-10c19e54b19c" (UID: "6fcb2820-6a23-4b19-9d23-10c19e54b19c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:44:33.228827 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.228809 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-envoy" pod "6fcb2820-6a23-4b19-9d23-10c19e54b19c" (UID: "6fcb2820-6a23-4b19-9d23-10c19e54b19c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:44:33.228887 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.228843 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-podinfo" pod "6fcb2820-6a23-4b19-9d23-10c19e54b19c" (UID: "6fcb2820-6a23-4b19-9d23-10c19e54b19c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:44:33.328114 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.328046 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8ll2\" (UniqueName: \"kubernetes.io/projected/6fcb2820-6a23-4b19-9d23-10c19e54b19c-kube-api-access-p8ll2\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:44:33.328114 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.328071 4947 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-token\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:44:33.328114 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.328080 4947 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-envoy\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:44:33.328114 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.328088 4947 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fcb2820-6a23-4b19-9d23-10c19e54b19c-istio-podinfo\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:44:33.972005 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.971965 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" event={"ID":"6fcb2820-6a23-4b19-9d23-10c19e54b19c","Type":"ContainerDied","Data":"3e0956b57472d1e39739d2957f7592ad2247b4b4965e901eb6329f19f40508ca"} Nov 03 17:44:33.972557 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.972036 4947 scope.go:122] "RemoveContainer" containerID="7b5122784ffc8bfbccd76ad91f10a12670fc0ee23247788fffd5efbf395ce8b4" Nov 03 17:44:33.972557 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:33.971980 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64" Nov 03 17:44:34.045819 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:34.045685 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64"] Nov 03 17:44:34.059015 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:34.058984 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-4b1292-openshift-default-687f98c6fcndg64"] Nov 03 17:44:34.567477 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:34.567429 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fcb2820-6a23-4b19-9d23-10c19e54b19c" path="/var/lib/kubelet/pods/6fcb2820-6a23-4b19-9d23-10c19e54b19c/volumes" Nov 03 17:44:37.293724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:37.292784 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:44:49.374594 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.374554 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs"] Nov 03 17:44:49.375133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.375001 4947 cpu_manager.go:414] "containerMap: removing container" podUID="6111883c-34de-4de0-aa85-e4849e24e67a" containerName="maas-api" Nov 03 17:44:49.375133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.375017 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6111883c-34de-4de0-aa85-e4849e24e67a" containerName="maas-api" Nov 03 17:44:49.375133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.375040 4947 cpu_manager.go:414] "containerMap: removing container" podUID="6fcb2820-6a23-4b19-9d23-10c19e54b19c" containerName="istio-proxy" Nov 03 17:44:49.375133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.375046 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fcb2820-6a23-4b19-9d23-10c19e54b19c" containerName="istio-proxy" Nov 03 17:44:49.375133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.375058 4947 cpu_manager.go:414] "containerMap: removing container" podUID="6fcb2820-6a23-4b19-9d23-10c19e54b19c" containerName="istio-proxy" Nov 03 17:44:49.375133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.375063 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fcb2820-6a23-4b19-9d23-10c19e54b19c" containerName="istio-proxy" Nov 03 17:44:49.375133 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.375132 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fcb2820-6a23-4b19-9d23-10c19e54b19c" containerName="istio-proxy" Nov 03 17:44:49.375557 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.375146 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6111883c-34de-4de0-aa85-e4849e24e67a" containerName="maas-api" Nov 03 17:44:49.379200 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.379181 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.399625 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.399595 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs"] Nov 03 17:44:49.410964 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.410942 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-4749b850-kserve-self-signed-certs\"" Nov 03 17:44:49.460063 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.460033 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-home\") pod \"e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.460224 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.460069 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-tmp-dir\") pod \"e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.460224 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.460089 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np776\" (UniqueName: \"kubernetes.io/projected/3c9f60bb-7a07-4040-9697-bf339f732cfa-kube-api-access-np776\") pod \"e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.460224 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.460108 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-model-cache\") pod \"e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.460348 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.460232 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9f60bb-7a07-4040-9697-bf339f732cfa-tls-certs\") pod \"e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.460348 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.460286 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-dshm\") pod \"e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.561781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.561739 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-dshm\") pod \"e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.562011 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.561910 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-home\") pod \"e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.562011 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.561941 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-tmp-dir\") pod \"e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.562011 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.561966 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np776\" (UniqueName: \"kubernetes.io/projected/3c9f60bb-7a07-4040-9697-bf339f732cfa-kube-api-access-np776\") pod \"e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.562011 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.562000 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-model-cache\") pod \"e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.562285 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.562043 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9f60bb-7a07-4040-9697-bf339f732cfa-tls-certs\") pod \"e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.562966 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.562938 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-tmp-dir\") pod \"e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.563300 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.563273 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-home\") pod \"e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.563517 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.563498 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-model-cache\") pod \"e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.574744 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.574667 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-dshm\") pod \"e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.575495 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.575116 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9f60bb-7a07-4040-9697-bf339f732cfa-tls-certs\") pod \"e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.592858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.592785 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np776\" (UniqueName: \"kubernetes.io/projected/3c9f60bb-7a07-4040-9697-bf339f732cfa-kube-api-access-np776\") pod \"e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:49.704947 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:49.704860 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:50.005774 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:50.005750 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs"] Nov 03 17:44:50.006936 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:44:50.006908 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c9f60bb_7a07_4040_9697_bf339f732cfa.slice/crio-2d525cf729525a8572b66f65445742344d20b4f12ebdb206ec656a110594acd0 WatchSource:0}: Error finding container 2d525cf729525a8572b66f65445742344d20b4f12ebdb206ec656a110594acd0: Status 404 returned error can't find the container with id 2d525cf729525a8572b66f65445742344d20b4f12ebdb206ec656a110594acd0 Nov 03 17:44:50.073109 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:50.073071 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" event={"ID":"3c9f60bb-7a07-4040-9697-bf339f732cfa","Type":"ContainerStarted","Data":"004d5e4a4af0eb8a9d7c12409b1ec5bfe8918e7cdc982d651bfc549a7ffa8a03"} Nov 03 17:44:50.073269 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:50.073123 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" event={"ID":"3c9f60bb-7a07-4040-9697-bf339f732cfa","Type":"ContainerStarted","Data":"2d525cf729525a8572b66f65445742344d20b4f12ebdb206ec656a110594acd0"} Nov 03 17:44:50.118483 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:50.118432 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" podStartSLOduration=1.118417163 podStartE2EDuration="1.118417163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:50.117688478 +0000 UTC m=+5433.706688447" watchObservedRunningTime="2026-11-03 17:44:50.118417163 +0000 UTC m=+5433.707417133" Nov 03 17:44:50.704954 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:50.704917 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:51.696896 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:51.696858 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:44:59.705567 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:59.705477 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:44:59.719108 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:44:59.719078 4947 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:45:00.129915 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:45:00.129872 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:45:00.162120 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:00.162090 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:45:07.751499 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:07.751463 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:45:18.767357 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:18.767319 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:45:23.050842 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:23.050809 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs"] Nov 03 17:45:26.246976 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:26.246944 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:45:30.129529 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:45:30.129487 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:45:38.072097 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.072040 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" podUID="3c9f60bb-7a07-4040-9697-bf339f732cfa" containerName="main" containerID="cri-o://004d5e4a4af0eb8a9d7c12409b1ec5bfe8918e7cdc982d651bfc549a7ffa8a03" gracePeriod=45 Nov 03 17:45:38.084956 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.084924 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:45:38.151088 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.151054 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" podUID="3c9f60bb-7a07-4040-9697-bf339f732cfa" containerName="main" probeResult="failure" output="Get \"https://10.132.1.38:8000/ready\": dial tcp 10.132.1.38:8000: connect: connection refused" Nov 03 17:45:38.336904 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.336841 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:45:38.356095 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.356071 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-dshm\") pod \"3c9f60bb-7a07-4040-9697-bf339f732cfa\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " Nov 03 17:45:38.356281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.356110 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-model-cache\") pod \"3c9f60bb-7a07-4040-9697-bf339f732cfa\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " Nov 03 17:45:38.356281 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.356145 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9f60bb-7a07-4040-9697-bf339f732cfa-kube-api-access-np776\" (UniqueName: \"kubernetes.io/projected/3c9f60bb-7a07-4040-9697-bf339f732cfa-kube-api-access-np776\") pod \"3c9f60bb-7a07-4040-9697-bf339f732cfa\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " Nov 03 17:45:38.356400 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.356337 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-home\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-home\") pod \"3c9f60bb-7a07-4040-9697-bf339f732cfa\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " Nov 03 17:45:38.356400 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.356373 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-tmp-dir\") pod \"3c9f60bb-7a07-4040-9697-bf339f732cfa\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " Nov 03 17:45:38.356510 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.356419 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-model-cache" pod "3c9f60bb-7a07-4040-9697-bf339f732cfa" (UID: "3c9f60bb-7a07-4040-9697-bf339f732cfa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:45:38.356510 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.356434 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c9f60bb-7a07-4040-9697-bf339f732cfa-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9f60bb-7a07-4040-9697-bf339f732cfa-tls-certs\") pod \"3c9f60bb-7a07-4040-9697-bf339f732cfa\" (UID: \"3c9f60bb-7a07-4040-9697-bf339f732cfa\") " Nov 03 17:45:38.356665 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.356646 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-tmp-dir" pod "3c9f60bb-7a07-4040-9697-bf339f732cfa" (UID: "3c9f60bb-7a07-4040-9697-bf339f732cfa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:45:38.356747 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.356687 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-home" pod "3c9f60bb-7a07-4040-9697-bf339f732cfa" (UID: "3c9f60bb-7a07-4040-9697-bf339f732cfa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:45:38.356863 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.356823 4947 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-home\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:45:38.356863 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.356845 4947 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-tmp-dir\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:45:38.356863 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.356859 4947 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-model-cache\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:45:38.358600 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.358571 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-dshm" pod "3c9f60bb-7a07-4040-9697-bf339f732cfa" (UID: "3c9f60bb-7a07-4040-9697-bf339f732cfa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:45:38.358773 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.358630 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9f60bb-7a07-4040-9697-bf339f732cfa-kube-api-access-np776" pod "3c9f60bb-7a07-4040-9697-bf339f732cfa" (UID: "3c9f60bb-7a07-4040-9697-bf339f732cfa"). InnerVolumeSpecName "kube-api-access-np776". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:45:38.358773 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.358661 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c9f60bb-7a07-4040-9697-bf339f732cfa-tls-certs" pod "3c9f60bb-7a07-4040-9697-bf339f732cfa" (UID: "3c9f60bb-7a07-4040-9697-bf339f732cfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:45:38.457616 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.457582 4947 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c9f60bb-7a07-4040-9697-bf339f732cfa-tls-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:45:38.457616 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.457609 4947 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c9f60bb-7a07-4040-9697-bf339f732cfa-dshm\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:45:38.457616 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.457620 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np776\" (UniqueName: \"kubernetes.io/projected/3c9f60bb-7a07-4040-9697-bf339f732cfa-kube-api-access-np776\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:45:38.549864 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.549827 4947 generic.go:356] "Generic (PLEG): container finished" podID="3c9f60bb-7a07-4040-9697-bf339f732cfa" containerID="004d5e4a4af0eb8a9d7c12409b1ec5bfe8918e7cdc982d651bfc549a7ffa8a03" exitCode=0 Nov 03 17:45:38.550132 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.549876 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" Nov 03 17:45:38.550282 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.549879 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" event={"ID":"3c9f60bb-7a07-4040-9697-bf339f732cfa","Type":"ContainerDied","Data":"004d5e4a4af0eb8a9d7c12409b1ec5bfe8918e7cdc982d651bfc549a7ffa8a03"} Nov 03 17:45:38.550282 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.550230 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs" event={"ID":"3c9f60bb-7a07-4040-9697-bf339f732cfa","Type":"ContainerDied","Data":"2d525cf729525a8572b66f65445742344d20b4f12ebdb206ec656a110594acd0"} Nov 03 17:45:38.550282 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.550255 4947 scope.go:122] "RemoveContainer" containerID="004d5e4a4af0eb8a9d7c12409b1ec5bfe8918e7cdc982d651bfc549a7ffa8a03" Nov 03 17:45:38.588295 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.588268 4947 scope.go:122] "RemoveContainer" containerID="004d5e4a4af0eb8a9d7c12409b1ec5bfe8918e7cdc982d651bfc549a7ffa8a03" Nov 03 17:45:38.589084 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:45:38.589053 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"004d5e4a4af0eb8a9d7c12409b1ec5bfe8918e7cdc982d651bfc549a7ffa8a03\": container with ID starting with 004d5e4a4af0eb8a9d7c12409b1ec5bfe8918e7cdc982d651bfc549a7ffa8a03 not found: ID does not exist" containerID="004d5e4a4af0eb8a9d7c12409b1ec5bfe8918e7cdc982d651bfc549a7ffa8a03" Nov 03 17:45:38.589174 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.589097 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004d5e4a4af0eb8a9d7c12409b1ec5bfe8918e7cdc982d651bfc549a7ffa8a03"} err="failed to get container status \"004d5e4a4af0eb8a9d7c12409b1ec5bfe8918e7cdc982d651bfc549a7ffa8a03\": rpc error: code = NotFound desc = could not find container \"004d5e4a4af0eb8a9d7c12409b1ec5bfe8918e7cdc982d651bfc549a7ffa8a03\": container with ID starting with 004d5e4a4af0eb8a9d7c12409b1ec5bfe8918e7cdc982d651bfc549a7ffa8a03 not found: ID does not exist" Nov 03 17:45:38.643720 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.642957 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs"] Nov 03 17:45:38.664723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:38.659740 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-4749b850-kserve-6d4cfb4889-j4sqs"] Nov 03 17:45:40.566391 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:40.566360 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c9f60bb-7a07-4040-9697-bf339f732cfa" path="/var/lib/kubelet/pods/3c9f60bb-7a07-4040-9697-bf339f732cfa/volumes" Nov 03 17:45:44.423724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:44.422859 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:45:44.423724 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:44.422926 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" podUID="3b61ea26-243b-4e8a-8ca7-3a154a72a9b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:45:45.386872 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:45.386829 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:45:57.364808 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:45:57.364768 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:46:00.130143 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:46:00.130102 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:46:29.877765 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:29.877663 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-8544d8cd96-nndq2"] Nov 03 17:46:29.878196 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:29.878134 4947 cpu_manager.go:414] "containerMap: removing container" podUID="3c9f60bb-7a07-4040-9697-bf339f732cfa" containerName="main" Nov 03 17:46:29.878196 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:29.878150 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c9f60bb-7a07-4040-9697-bf339f732cfa" containerName="main" Nov 03 17:46:29.878279 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:29.878226 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fcb2820-6a23-4b19-9d23-10c19e54b19c" containerName="istio-proxy" Nov 03 17:46:29.878279 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:29.878239 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c9f60bb-7a07-4040-9697-bf339f732cfa" containerName="main" Nov 03 17:46:29.882673 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:29.882650 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-8544d8cd96-nndq2" Nov 03 17:46:29.902882 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:29.902854 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-8544d8cd96-nndq2"] Nov 03 17:46:29.914279 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:29.914251 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 17:46:29.914456 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:29.914359 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-qp7nb\"" Nov 03 17:46:29.972403 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:29.972362 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z596p\" (UniqueName: \"kubernetes.io/projected/ca409853-f894-42a7-b25b-5731b743cf8d-kube-api-access-z596p\") pod \"payload-pre-processing-8544d8cd96-nndq2\" (UID: \"ca409853-f894-42a7-b25b-5731b743cf8d\") " pod="openshift-ingress/payload-pre-processing-8544d8cd96-nndq2" Nov 03 17:46:29.972582 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:29.972422 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca409853-f894-42a7-b25b-5731b743cf8d-plugins-config-volume\") pod \"payload-pre-processing-8544d8cd96-nndq2\" (UID: \"ca409853-f894-42a7-b25b-5731b743cf8d\") " pod="openshift-ingress/payload-pre-processing-8544d8cd96-nndq2" Nov 03 17:46:30.073239 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:30.073206 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z596p\" (UniqueName: \"kubernetes.io/projected/ca409853-f894-42a7-b25b-5731b743cf8d-kube-api-access-z596p\") pod \"payload-pre-processing-8544d8cd96-nndq2\" (UID: \"ca409853-f894-42a7-b25b-5731b743cf8d\") " pod="openshift-ingress/payload-pre-processing-8544d8cd96-nndq2" Nov 03 17:46:30.073452 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:30.073265 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca409853-f894-42a7-b25b-5731b743cf8d-plugins-config-volume\") pod \"payload-pre-processing-8544d8cd96-nndq2\" (UID: \"ca409853-f894-42a7-b25b-5731b743cf8d\") " pod="openshift-ingress/payload-pre-processing-8544d8cd96-nndq2" Nov 03 17:46:30.074022 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:30.074000 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca409853-f894-42a7-b25b-5731b743cf8d-plugins-config-volume\") pod \"payload-pre-processing-8544d8cd96-nndq2\" (UID: \"ca409853-f894-42a7-b25b-5731b743cf8d\") " pod="openshift-ingress/payload-pre-processing-8544d8cd96-nndq2" Nov 03 17:46:30.098969 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:30.098936 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z596p\" (UniqueName: \"kubernetes.io/projected/ca409853-f894-42a7-b25b-5731b743cf8d-kube-api-access-z596p\") pod \"payload-pre-processing-8544d8cd96-nndq2\" (UID: \"ca409853-f894-42a7-b25b-5731b743cf8d\") " pod="openshift-ingress/payload-pre-processing-8544d8cd96-nndq2" Nov 03 17:46:30.129400 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:46:30.129326 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:46:30.195121 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:30.195094 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-8544d8cd96-nndq2" Nov 03 17:46:30.494904 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:30.494877 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-8544d8cd96-nndq2"] Nov 03 17:46:30.496278 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:46:30.496243 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca409853_f894_42a7_b25b_5731b743cf8d.slice/crio-b7c8a807fd2de80d9655dd0e57ca1e06d650c6a82386aa3af70795b62a9fb6b7 WatchSource:0}: Error finding container b7c8a807fd2de80d9655dd0e57ca1e06d650c6a82386aa3af70795b62a9fb6b7: Status 404 returned error can't find the container with id b7c8a807fd2de80d9655dd0e57ca1e06d650c6a82386aa3af70795b62a9fb6b7 Nov 03 17:46:30.863657 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:30.863571 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-8544d8cd96-nndq2" event={"ID":"ca409853-f894-42a7-b25b-5731b743cf8d","Type":"ContainerStarted","Data":"a8fffe5d101644cdfb28de6294db8d3d18f5df5c28a8a4d3c3c27575d787cef6"} Nov 03 17:46:30.863878 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:30.863729 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-8544d8cd96-nndq2" Nov 03 17:46:30.863878 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:30.863774 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-8544d8cd96-nndq2" event={"ID":"ca409853-f894-42a7-b25b-5731b743cf8d","Type":"ContainerStarted","Data":"b7c8a807fd2de80d9655dd0e57ca1e06d650c6a82386aa3af70795b62a9fb6b7"} Nov 03 17:46:30.908314 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:30.908266 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-8544d8cd96-nndq2" podStartSLOduration=1.90825216 podStartE2EDuration="1.90825216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:30.907770778 +0000 UTC m=+5534.496770759" watchObservedRunningTime="2026-11-03 17:46:30.90825216 +0000 UTC m=+5534.497252126" Nov 03 17:46:41.872310 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:41.872278 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-8544d8cd96-nndq2" Nov 03 17:46:46.490150 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:46:46.490104 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:47:00.128342 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:47:00.128302 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:47:30.131462 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:47:30.131419 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:48:00.129994 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:48:00.129903 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:48:26.283431 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:48:26.283397 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:48:30.129027 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:48:30.128991 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:48:33.346451 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:48:33.346417 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:48:49.242773 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:48:49.242736 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:48:53.889249 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:48:53.889189 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:49:00.128848 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:49:00.128810 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:49:09.059653 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:49:09.059611 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:49:14.427047 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:49:14.426996 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:49:14.427767 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:49:14.427727 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" podUID="3b61ea26-243b-4e8a-8ca7-3a154a72a9b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:49:17.447891 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:49:17.447842 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:49:17.460152 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:49:17.460124 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:49:24.566896 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:49:24.566862 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:49:30.129929 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:49:30.129839 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:49:47.757453 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:49:47.757419 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:49:58.068416 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:49:58.068374 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:50:00.130445 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:50:00.130404 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:50:02.660781 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:50:02.660743 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:50:19.472839 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:50:19.472804 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:50:24.643962 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:50:24.643924 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:50:28.864492 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:50:28.864459 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:50:30.129364 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:50:30.129328 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:50:40.789222 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:50:40.789188 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:50:45.891156 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:50:45.891124 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:50:54.294635 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:50:54.294602 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:50:59.461346 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:50:59.461266 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:51:00.129407 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:51:00.129362 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:51:17.787743 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:51:17.787706 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:51:23.633770 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:51:23.632988 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:51:30.129369 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:51:30.129326 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:51:36.591481 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:51:36.591440 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:51:41.976246 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:51:41.976211 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:51:48.582447 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:51:48.582416 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:52:00.129470 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:52:00.129432 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:52:06.289573 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:06.289525 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:52:16.657622 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:16.657587 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:52:30.130190 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:52:30.130103 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:52:34.157522 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:34.157478 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:52:40.796903 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:40.796867 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:52:43.139224 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:43.139188 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb"] Nov 03 17:52:43.143124 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:43.143103 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:52:43.177577 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:43.177534 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-27d3f0-serving-cert\"" Nov 03 17:52:43.181599 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:43.181571 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb"] Nov 03 17:52:43.214741 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:43.214711 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e03bcc30-aabf-4914-9b63-e48ad597db9a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb\" (UID: \"e03bcc30-aabf-4914-9b63-e48ad597db9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:52:43.214906 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:43.214772 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn2hc\" (UniqueName: \"kubernetes.io/projected/e03bcc30-aabf-4914-9b63-e48ad597db9a-kube-api-access-kn2hc\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb\" (UID: \"e03bcc30-aabf-4914-9b63-e48ad597db9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:52:43.215018 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:43.214895 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e03bcc30-aabf-4914-9b63-e48ad597db9a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb\" (UID: \"e03bcc30-aabf-4914-9b63-e48ad597db9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:52:43.215018 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:43.214939 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03bcc30-aabf-4914-9b63-e48ad597db9a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb\" (UID: \"e03bcc30-aabf-4914-9b63-e48ad597db9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:52:43.315639 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:43.315591 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03bcc30-aabf-4914-9b63-e48ad597db9a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb\" (UID: \"e03bcc30-aabf-4914-9b63-e48ad597db9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:52:43.315863 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:43.315674 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e03bcc30-aabf-4914-9b63-e48ad597db9a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb\" (UID: \"e03bcc30-aabf-4914-9b63-e48ad597db9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:52:43.315863 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:43.315718 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn2hc\" (UniqueName: \"kubernetes.io/projected/e03bcc30-aabf-4914-9b63-e48ad597db9a-kube-api-access-kn2hc\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb\" (UID: \"e03bcc30-aabf-4914-9b63-e48ad597db9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:52:43.315863 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:43.315799 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e03bcc30-aabf-4914-9b63-e48ad597db9a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb\" (UID: \"e03bcc30-aabf-4914-9b63-e48ad597db9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:52:43.315863 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:52:43.315827 4947 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-serving-cert: secret "maas-api-e2e-worker-single-main-27d3f0-serving-cert" not found Nov 03 17:52:43.316122 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:52:43.315950 4947 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e03bcc30-aabf-4914-9b63-e48ad597db9a-maas-api-tls podName:e03bcc30-aabf-4914-9b63-e48ad597db9a nodeName:}" failed. No retries permitted until 2026-11-03 17:52:43.815926325 +0000 UTC m=+5907.404926274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e03bcc30-aabf-4914-9b63-e48ad597db9a-maas-api-tls") pod "maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" (UID: "e03bcc30-aabf-4914-9b63-e48ad597db9a") : secret "maas-api-e2e-worker-single-main-27d3f0-serving-cert" not found Nov 03 17:52:43.316443 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:43.316420 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03bcc30-aabf-4914-9b63-e48ad597db9a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb\" (UID: \"e03bcc30-aabf-4914-9b63-e48ad597db9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:52:43.318732 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:43.318679 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e03bcc30-aabf-4914-9b63-e48ad597db9a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb\" (UID: \"e03bcc30-aabf-4914-9b63-e48ad597db9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:52:43.352080 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:43.352043 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn2hc\" (UniqueName: \"kubernetes.io/projected/e03bcc30-aabf-4914-9b63-e48ad597db9a-kube-api-access-kn2hc\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb\" (UID: \"e03bcc30-aabf-4914-9b63-e48ad597db9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:52:43.824673 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:43.824631 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e03bcc30-aabf-4914-9b63-e48ad597db9a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb\" (UID: \"e03bcc30-aabf-4914-9b63-e48ad597db9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:52:43.827384 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:43.827345 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e03bcc30-aabf-4914-9b63-e48ad597db9a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb\" (UID: \"e03bcc30-aabf-4914-9b63-e48ad597db9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:52:44.057252 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:44.057218 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:52:44.365460 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:44.365360 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb"] Nov 03 17:52:44.370871 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:44.370847 4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:52:45.103246 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:45.103162 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" event={"ID":"e03bcc30-aabf-4914-9b63-e48ad597db9a","Type":"ContainerStarted","Data":"a6f4c21561ecb5871514d85bc0e518b71e1bd93c6360040758809c47914d695d"} Nov 03 17:52:45.103246 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:45.103212 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" event={"ID":"e03bcc30-aabf-4914-9b63-e48ad597db9a","Type":"ContainerStarted","Data":"04fec1bb4f5b8f0f5837b96487c7bc996c7703357e61a9b7f6078dbd2a07c914"} Nov 03 17:52:45.103439 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:45.103269 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:52:45.150053 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:45.149999 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" podStartSLOduration=1.705291493 podStartE2EDuration="2.149983618s" totalImagesPullingTime="444.692125ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:45.148343014 +0000 UTC m=+5908.737342984" watchObservedRunningTime="2026-11-03 17:52:45.149983618 +0000 UTC m=+5908.738983636" Nov 03 17:52:46.871916 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:46.871877 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:52:51.113910 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:52:51.113875 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:53:00.129576 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:53:00.129532 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:53:01.670886 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:01.670850 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:53:16.646969 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:16.646936 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:53:30.130108 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:53:30.130068 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:53:42.693040 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:42.693006 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:53:49.530583 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:49.530546 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb"] Nov 03 17:53:49.531252 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:49.530990 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" podUID="e03bcc30-aabf-4914-9b63-e48ad597db9a" containerName="maas-api" containerID="cri-o://a6f4c21561ecb5871514d85bc0e518b71e1bd93c6360040758809c47914d695d" gracePeriod=30 Nov 03 17:53:50.018325 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.018296 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:53:50.139244 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.139165 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e03bcc30-aabf-4914-9b63-e48ad597db9a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e03bcc30-aabf-4914-9b63-e48ad597db9a-maas-api-tls\") pod \"e03bcc30-aabf-4914-9b63-e48ad597db9a\" (UID: \"e03bcc30-aabf-4914-9b63-e48ad597db9a\") " Nov 03 17:53:50.139244 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.139232 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e03bcc30-aabf-4914-9b63-e48ad597db9a-kube-api-access-kn2hc\" (UniqueName: \"kubernetes.io/projected/e03bcc30-aabf-4914-9b63-e48ad597db9a-kube-api-access-kn2hc\") pod \"e03bcc30-aabf-4914-9b63-e48ad597db9a\" (UID: \"e03bcc30-aabf-4914-9b63-e48ad597db9a\") " Nov 03 17:53:50.139442 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.139337 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e03bcc30-aabf-4914-9b63-e48ad597db9a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03bcc30-aabf-4914-9b63-e48ad597db9a-openshift-service-ca\") pod \"e03bcc30-aabf-4914-9b63-e48ad597db9a\" (UID: \"e03bcc30-aabf-4914-9b63-e48ad597db9a\") " Nov 03 17:53:50.139481 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.139441 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e03bcc30-aabf-4914-9b63-e48ad597db9a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e03bcc30-aabf-4914-9b63-e48ad597db9a-metrics-certs\") pod \"e03bcc30-aabf-4914-9b63-e48ad597db9a\" (UID: \"e03bcc30-aabf-4914-9b63-e48ad597db9a\") " Nov 03 17:53:50.139689 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.139667 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e03bcc30-aabf-4914-9b63-e48ad597db9a-openshift-service-ca" pod "e03bcc30-aabf-4914-9b63-e48ad597db9a" (UID: "e03bcc30-aabf-4914-9b63-e48ad597db9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:53:50.139859 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.139843 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03bcc30-aabf-4914-9b63-e48ad597db9a-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:53:50.141637 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.141603 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e03bcc30-aabf-4914-9b63-e48ad597db9a-metrics-certs" pod "e03bcc30-aabf-4914-9b63-e48ad597db9a" (UID: "e03bcc30-aabf-4914-9b63-e48ad597db9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:53:50.141637 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.141632 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e03bcc30-aabf-4914-9b63-e48ad597db9a-maas-api-tls" pod "e03bcc30-aabf-4914-9b63-e48ad597db9a" (UID: "e03bcc30-aabf-4914-9b63-e48ad597db9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:53:50.141835 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.141685 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e03bcc30-aabf-4914-9b63-e48ad597db9a-kube-api-access-kn2hc" pod "e03bcc30-aabf-4914-9b63-e48ad597db9a" (UID: "e03bcc30-aabf-4914-9b63-e48ad597db9a"). InnerVolumeSpecName "kube-api-access-kn2hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:53:50.241328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.241298 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e03bcc30-aabf-4914-9b63-e48ad597db9a-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:53:50.241328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.241324 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e03bcc30-aabf-4914-9b63-e48ad597db9a-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:53:50.241328 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.241333 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn2hc\" (UniqueName: \"kubernetes.io/projected/e03bcc30-aabf-4914-9b63-e48ad597db9a-kube-api-access-kn2hc\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:53:50.421290 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.421189 4947 generic.go:356] "Generic (PLEG): container finished" podID="e03bcc30-aabf-4914-9b63-e48ad597db9a" containerID="a6f4c21561ecb5871514d85bc0e518b71e1bd93c6360040758809c47914d695d" exitCode=0 Nov 03 17:53:50.421290 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.421242 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" event={"ID":"e03bcc30-aabf-4914-9b63-e48ad597db9a","Type":"ContainerDied","Data":"a6f4c21561ecb5871514d85bc0e518b71e1bd93c6360040758809c47914d695d"} Nov 03 17:53:50.421290 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.421246 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" Nov 03 17:53:50.421290 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.421283 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb" event={"ID":"e03bcc30-aabf-4914-9b63-e48ad597db9a","Type":"ContainerDied","Data":"04fec1bb4f5b8f0f5837b96487c7bc996c7703357e61a9b7f6078dbd2a07c914"} Nov 03 17:53:50.421656 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.421304 4947 scope.go:122] "RemoveContainer" containerID="a6f4c21561ecb5871514d85bc0e518b71e1bd93c6360040758809c47914d695d" Nov 03 17:53:50.430972 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.430953 4947 scope.go:122] "RemoveContainer" containerID="a6f4c21561ecb5871514d85bc0e518b71e1bd93c6360040758809c47914d695d" Nov 03 17:53:50.431224 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:53:50.431205 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6f4c21561ecb5871514d85bc0e518b71e1bd93c6360040758809c47914d695d\": container with ID starting with a6f4c21561ecb5871514d85bc0e518b71e1bd93c6360040758809c47914d695d not found: ID does not exist" containerID="a6f4c21561ecb5871514d85bc0e518b71e1bd93c6360040758809c47914d695d" Nov 03 17:53:50.431271 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.431234 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f4c21561ecb5871514d85bc0e518b71e1bd93c6360040758809c47914d695d"} err="failed to get container status \"a6f4c21561ecb5871514d85bc0e518b71e1bd93c6360040758809c47914d695d\": rpc error: code = NotFound desc = could not find container \"a6f4c21561ecb5871514d85bc0e518b71e1bd93c6360040758809c47914d695d\": container with ID starting with a6f4c21561ecb5871514d85bc0e518b71e1bd93c6360040758809c47914d695d not found: ID does not exist" Nov 03 17:53:50.476115 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.476039 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb"] Nov 03 17:53:50.489306 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.489280 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-l7xgb"] Nov 03 17:53:50.563737 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:50.563686 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e03bcc30-aabf-4914-9b63-e48ad597db9a" path="/var/lib/kubelet/pods/e03bcc30-aabf-4914-9b63-e48ad597db9a/volumes" Nov 03 17:53:51.943096 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:53:51.943060 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:54:00.129455 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:54:00.129372 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:54:17.480460 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:17.480428 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:54:17.490081 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:17.490058 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:54:21.632858 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.632827 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn"] Nov 03 17:54:21.633514 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.633450 4947 cpu_manager.go:414] "containerMap: removing container" podUID="e03bcc30-aabf-4914-9b63-e48ad597db9a" containerName="maas-api" Nov 03 17:54:21.633514 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.633471 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e03bcc30-aabf-4914-9b63-e48ad597db9a" containerName="maas-api" Nov 03 17:54:21.633653 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.633603 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="e03bcc30-aabf-4914-9b63-e48ad597db9a" containerName="maas-api" Nov 03 17:54:21.637865 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.637841 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" Nov 03 17:54:21.669618 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.669585 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-27d3f0-serving-cert\"" Nov 03 17:54:21.672632 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.672589 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn"] Nov 03 17:54:21.764989 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.764943 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n42bq\" (UniqueName: \"kubernetes.io/projected/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-kube-api-access-n42bq\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn\" (UID: \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" Nov 03 17:54:21.765194 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.765086 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn\" (UID: \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" Nov 03 17:54:21.765194 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.765132 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn\" (UID: \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" Nov 03 17:54:21.765194 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.765160 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn\" (UID: \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" Nov 03 17:54:21.866090 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.866055 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn\" (UID: \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" Nov 03 17:54:21.866304 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.866108 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn\" (UID: \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" Nov 03 17:54:21.866304 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.866134 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn\" (UID: \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" Nov 03 17:54:21.866304 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.866178 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n42bq\" (UniqueName: \"kubernetes.io/projected/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-kube-api-access-n42bq\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn\" (UID: \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" Nov 03 17:54:21.866764 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.866730 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn\" (UID: \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" Nov 03 17:54:21.868745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.868687 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn\" (UID: \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" Nov 03 17:54:21.868893 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.868865 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn\" (UID: \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" Nov 03 17:54:21.899006 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.898932 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n42bq\" (UniqueName: \"kubernetes.io/projected/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-kube-api-access-n42bq\") pod \"maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn\" (UID: \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" Nov 03 17:54:21.950893 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:21.950853 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" Nov 03 17:54:22.251645 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:22.251614 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn"] Nov 03 17:54:22.252417 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:54:22.252385 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f74855_ccf3_4adc_a87e_c810ef5ce5c1.slice/crio-69df746c6e07df9a8fb3cb9f9de895b17ad20dc602febdbdd4d281401fb9591c WatchSource:0}: Error finding container 69df746c6e07df9a8fb3cb9f9de895b17ad20dc602febdbdd4d281401fb9591c: Status 404 returned error can't find the container with id 69df746c6e07df9a8fb3cb9f9de895b17ad20dc602febdbdd4d281401fb9591c Nov 03 17:54:22.572912 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:22.572834 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" event={"ID":"18f74855-ccf3-4adc-a87e-c810ef5ce5c1","Type":"ContainerStarted","Data":"69df746c6e07df9a8fb3cb9f9de895b17ad20dc602febdbdd4d281401fb9591c"} Nov 03 17:54:23.578924 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:23.578886 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" event={"ID":"18f74855-ccf3-4adc-a87e-c810ef5ce5c1","Type":"ContainerStarted","Data":"64bcd2d32cf4d20447389bf64dfc2c5fbaffb007712f58c8c159216d7de04564"} Nov 03 17:54:23.579340 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:23.579014 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" Nov 03 17:54:23.626273 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:23.626219 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" podStartSLOduration=2.305944854 podStartE2EDuration="2.62620551s" totalImagesPullingTime="320.260656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:23.625777007 +0000 UTC m=+6007.214776976" watchObservedRunningTime="2026-11-03 17:54:23.62620551 +0000 UTC m=+6007.215205478" Nov 03 17:54:25.683632 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:25.683593 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:54:29.589923 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:29.589892 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" Nov 03 17:54:30.129466 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:54:30.129428 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:54:42.247445 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.247406 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn"] Nov 03 17:54:42.247951 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.247669 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" podUID="18f74855-ccf3-4adc-a87e-c810ef5ce5c1" containerName="maas-api" containerID="cri-o://64bcd2d32cf4d20447389bf64dfc2c5fbaffb007712f58c8c159216d7de04564" gracePeriod=30 Nov 03 17:54:42.529673 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.529630 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" Nov 03 17:54:42.610182 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.610150 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-openshift-service-ca\") pod \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\" (UID: \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\") " Nov 03 17:54:42.610372 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.610194 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-maas-api-tls\") pod \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\" (UID: \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\") " Nov 03 17:54:42.610372 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.610222 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-kube-api-access-n42bq\" (UniqueName: \"kubernetes.io/projected/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-kube-api-access-n42bq\") pod \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\" (UID: \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\") " Nov 03 17:54:42.610372 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.610285 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-metrics-certs\") pod \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\" (UID: \"18f74855-ccf3-4adc-a87e-c810ef5ce5c1\") " Nov 03 17:54:42.610632 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.610587 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-openshift-service-ca" pod "18f74855-ccf3-4adc-a87e-c810ef5ce5c1" (UID: "18f74855-ccf3-4adc-a87e-c810ef5ce5c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:54:42.612574 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.612540 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-maas-api-tls" pod "18f74855-ccf3-4adc-a87e-c810ef5ce5c1" (UID: "18f74855-ccf3-4adc-a87e-c810ef5ce5c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:54:42.612725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.612689 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-metrics-certs" pod "18f74855-ccf3-4adc-a87e-c810ef5ce5c1" (UID: "18f74855-ccf3-4adc-a87e-c810ef5ce5c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:54:42.612725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.612688 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-kube-api-access-n42bq" pod "18f74855-ccf3-4adc-a87e-c810ef5ce5c1" (UID: "18f74855-ccf3-4adc-a87e-c810ef5ce5c1"). InnerVolumeSpecName "kube-api-access-n42bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:54:42.713002 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.711223 4947 generic.go:356] "Generic (PLEG): container finished" podID="18f74855-ccf3-4adc-a87e-c810ef5ce5c1" containerID="64bcd2d32cf4d20447389bf64dfc2c5fbaffb007712f58c8c159216d7de04564" exitCode=0 Nov 03 17:54:42.713002 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.711306 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" event={"ID":"18f74855-ccf3-4adc-a87e-c810ef5ce5c1","Type":"ContainerDied","Data":"64bcd2d32cf4d20447389bf64dfc2c5fbaffb007712f58c8c159216d7de04564"} Nov 03 17:54:42.713002 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.711357 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" event={"ID":"18f74855-ccf3-4adc-a87e-c810ef5ce5c1","Type":"ContainerDied","Data":"69df746c6e07df9a8fb3cb9f9de895b17ad20dc602febdbdd4d281401fb9591c"} Nov 03 17:54:42.713002 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.711388 4947 scope.go:122] "RemoveContainer" containerID="64bcd2d32cf4d20447389bf64dfc2c5fbaffb007712f58c8c159216d7de04564" Nov 03 17:54:42.713002 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.711582 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn" Nov 03 17:54:42.717632 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.717509 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:54:42.717632 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.717569 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:54:42.717632 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.717585 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:54:42.717632 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.717597 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n42bq\" (UniqueName: \"kubernetes.io/projected/18f74855-ccf3-4adc-a87e-c810ef5ce5c1-kube-api-access-n42bq\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:54:42.742682 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.742659 4947 scope.go:122] "RemoveContainer" containerID="64bcd2d32cf4d20447389bf64dfc2c5fbaffb007712f58c8c159216d7de04564" Nov 03 17:54:42.743086 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:54:42.743066 4947 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64bcd2d32cf4d20447389bf64dfc2c5fbaffb007712f58c8c159216d7de04564\": container with ID starting with 64bcd2d32cf4d20447389bf64dfc2c5fbaffb007712f58c8c159216d7de04564 not found: ID does not exist" containerID="64bcd2d32cf4d20447389bf64dfc2c5fbaffb007712f58c8c159216d7de04564" Nov 03 17:54:42.743146 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.743097 4947 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64bcd2d32cf4d20447389bf64dfc2c5fbaffb007712f58c8c159216d7de04564"} err="failed to get container status \"64bcd2d32cf4d20447389bf64dfc2c5fbaffb007712f58c8c159216d7de04564\": rpc error: code = NotFound desc = could not find container \"64bcd2d32cf4d20447389bf64dfc2c5fbaffb007712f58c8c159216d7de04564\": container with ID starting with 64bcd2d32cf4d20447389bf64dfc2c5fbaffb007712f58c8c159216d7de04564 not found: ID does not exist" Nov 03 17:54:42.775624 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.775550 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn"] Nov 03 17:54:42.777153 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:42.777133 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-27d3f0-76b6dbd9b8-xkkrn"] Nov 03 17:54:44.564515 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:44.564478 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18f74855-ccf3-4adc-a87e-c810ef5ce5c1" path="/var/lib/kubelet/pods/18f74855-ccf3-4adc-a87e-c810ef5ce5c1/volumes" Nov 03 17:54:45.298297 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:45.298260 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:54:56.655367 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:54:56.655337 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:55:00.129553 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:55:00.129515 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:55:30.129598 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:55:30.129511 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:56:00.129414 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:56:00.129370 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:56:25.029709 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:25.029673 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:56:30.128691 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:56:30.128656 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:56:40.492314 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:40.492267 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:56:45.793730 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:45.792869 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:56:45.801725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:45.797865 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mjl62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:56:45.801725 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:45.797935 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" podUID="37849d8a-115b-44a5-bea8-539a92f53653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:56:50.260016 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:50.259983 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:56:56.084020 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.083983 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft"] Nov 03 17:56:56.084566 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.084546 4947 cpu_manager.go:414] "containerMap: removing container" podUID="18f74855-ccf3-4adc-a87e-c810ef5ce5c1" containerName="maas-api" Nov 03 17:56:56.084566 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.084567 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18f74855-ccf3-4adc-a87e-c810ef5ce5c1" containerName="maas-api" Nov 03 17:56:56.084663 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.084637 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="18f74855-ccf3-4adc-a87e-c810ef5ce5c1" containerName="maas-api" Nov 03 17:56:56.087989 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.087968 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" Nov 03 17:56:56.119316 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.119279 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft"] Nov 03 17:56:56.170238 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.170202 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17bd3a7-aadb-444c-8835-8e21fdbe1343-openshift-service-ca\") pod \"maas-api-78d9c46c8c-lm2ft\" (UID: \"e17bd3a7-aadb-444c-8835-8e21fdbe1343\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" Nov 03 17:56:56.170523 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.170271 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwzh\" (UniqueName: \"kubernetes.io/projected/e17bd3a7-aadb-444c-8835-8e21fdbe1343-kube-api-access-xqwzh\") pod \"maas-api-78d9c46c8c-lm2ft\" (UID: \"e17bd3a7-aadb-444c-8835-8e21fdbe1343\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" Nov 03 17:56:56.170523 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.170356 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17bd3a7-aadb-444c-8835-8e21fdbe1343-metrics-certs\") pod \"maas-api-78d9c46c8c-lm2ft\" (UID: \"e17bd3a7-aadb-444c-8835-8e21fdbe1343\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" Nov 03 17:56:56.170523 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.170426 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e17bd3a7-aadb-444c-8835-8e21fdbe1343-maas-api-tls\") pod \"maas-api-78d9c46c8c-lm2ft\" (UID: \"e17bd3a7-aadb-444c-8835-8e21fdbe1343\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" Nov 03 17:56:56.271008 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.270974 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17bd3a7-aadb-444c-8835-8e21fdbe1343-openshift-service-ca\") pod \"maas-api-78d9c46c8c-lm2ft\" (UID: \"e17bd3a7-aadb-444c-8835-8e21fdbe1343\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" Nov 03 17:56:56.271179 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.271017 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwzh\" (UniqueName: \"kubernetes.io/projected/e17bd3a7-aadb-444c-8835-8e21fdbe1343-kube-api-access-xqwzh\") pod \"maas-api-78d9c46c8c-lm2ft\" (UID: \"e17bd3a7-aadb-444c-8835-8e21fdbe1343\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" Nov 03 17:56:56.271179 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.271044 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17bd3a7-aadb-444c-8835-8e21fdbe1343-metrics-certs\") pod \"maas-api-78d9c46c8c-lm2ft\" (UID: \"e17bd3a7-aadb-444c-8835-8e21fdbe1343\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" Nov 03 17:56:56.271179 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.271093 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e17bd3a7-aadb-444c-8835-8e21fdbe1343-maas-api-tls\") pod \"maas-api-78d9c46c8c-lm2ft\" (UID: \"e17bd3a7-aadb-444c-8835-8e21fdbe1343\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" Nov 03 17:56:56.271716 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.271671 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e17bd3a7-aadb-444c-8835-8e21fdbe1343-openshift-service-ca\") pod \"maas-api-78d9c46c8c-lm2ft\" (UID: \"e17bd3a7-aadb-444c-8835-8e21fdbe1343\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" Nov 03 17:56:56.273682 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.273660 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e17bd3a7-aadb-444c-8835-8e21fdbe1343-maas-api-tls\") pod \"maas-api-78d9c46c8c-lm2ft\" (UID: \"e17bd3a7-aadb-444c-8835-8e21fdbe1343\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" Nov 03 17:56:56.273827 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.273804 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e17bd3a7-aadb-444c-8835-8e21fdbe1343-metrics-certs\") pod \"maas-api-78d9c46c8c-lm2ft\" (UID: \"e17bd3a7-aadb-444c-8835-8e21fdbe1343\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" Nov 03 17:56:56.295542 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.295514 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwzh\" (UniqueName: \"kubernetes.io/projected/e17bd3a7-aadb-444c-8835-8e21fdbe1343-kube-api-access-xqwzh\") pod \"maas-api-78d9c46c8c-lm2ft\" (UID: \"e17bd3a7-aadb-444c-8835-8e21fdbe1343\") " pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" Nov 03 17:56:56.404837 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.404754 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" Nov 03 17:56:56.708400 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:56.708370 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft"] Nov 03 17:56:56.709837 ip-10-0-143-64 kubenswrapper[4947]: W1103 17:56:56.709807 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17bd3a7_aadb_444c_8835_8e21fdbe1343.slice/crio-2375652d2d322567111631c9033892ca9ec40debf7c1b33e86526a955dc719bf WatchSource:0}: Error finding container 2375652d2d322567111631c9033892ca9ec40debf7c1b33e86526a955dc719bf: Status 404 returned error can't find the container with id 2375652d2d322567111631c9033892ca9ec40debf7c1b33e86526a955dc719bf Nov 03 17:56:57.433335 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:57.433216 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" event={"ID":"e17bd3a7-aadb-444c-8835-8e21fdbe1343","Type":"ContainerStarted","Data":"4bfa5e25e9047914cca6280b6b22599c232d7bbb43a862f6c1dbe7a3d35f913a"} Nov 03 17:56:57.433335 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:57.433288 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" event={"ID":"e17bd3a7-aadb-444c-8835-8e21fdbe1343","Type":"ContainerStarted","Data":"2375652d2d322567111631c9033892ca9ec40debf7c1b33e86526a955dc719bf"} Nov 03 17:56:57.433335 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:57.433325 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" Nov 03 17:56:57.478952 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:56:57.478896 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" podStartSLOduration=1.120817059 podStartE2EDuration="1.478878316s" totalImagesPullingTime="358.061257ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:57.478225199 +0000 UTC m=+6161.067225168" watchObservedRunningTime="2026-11-03 17:56:57.478878316 +0000 UTC m=+6161.067878286" Nov 03 17:57:00.128875 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:57:00.128836 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:57:03.443546 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:03.443518 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78d9c46c8c-lm2ft" Nov 03 17:57:07.080514 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:07.080481 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:57:11.104160 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:11.104109 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk"] Nov 03 17:57:11.104778 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:11.104369 4947 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" podUID="9733bd5c-945d-4308-92b5-a838e82acf50" containerName="maas-api" containerID="cri-o://4f9640d12ee0c8b6b77c2ce4c2b84fa87169cf2a093680330b8358e14a37a7da" gracePeriod=30 Nov 03 17:57:12.624723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:12.620645 4947 generic.go:356] "Generic (PLEG): container finished" podID="9733bd5c-945d-4308-92b5-a838e82acf50" containerID="4f9640d12ee0c8b6b77c2ce4c2b84fa87169cf2a093680330b8358e14a37a7da" exitCode=0 Nov 03 17:57:12.663721 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:12.648803 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" event={"ID":"9733bd5c-945d-4308-92b5-a838e82acf50","Type":"ContainerDied","Data":"4f9640d12ee0c8b6b77c2ce4c2b84fa87169cf2a093680330b8358e14a37a7da"} Nov 03 17:57:13.100945 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:13.100911 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" Nov 03 17:57:13.248722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:13.247054 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9733bd5c-945d-4308-92b5-a838e82acf50-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9733bd5c-945d-4308-92b5-a838e82acf50-openshift-service-ca\") pod \"9733bd5c-945d-4308-92b5-a838e82acf50\" (UID: \"9733bd5c-945d-4308-92b5-a838e82acf50\") " Nov 03 17:57:13.248722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:13.247215 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9733bd5c-945d-4308-92b5-a838e82acf50-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9733bd5c-945d-4308-92b5-a838e82acf50-metrics-certs\") pod \"9733bd5c-945d-4308-92b5-a838e82acf50\" (UID: \"9733bd5c-945d-4308-92b5-a838e82acf50\") " Nov 03 17:57:13.248722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:13.247255 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9733bd5c-945d-4308-92b5-a838e82acf50-kube-api-access-2zrqv\" (UniqueName: \"kubernetes.io/projected/9733bd5c-945d-4308-92b5-a838e82acf50-kube-api-access-2zrqv\") pod \"9733bd5c-945d-4308-92b5-a838e82acf50\" (UID: \"9733bd5c-945d-4308-92b5-a838e82acf50\") " Nov 03 17:57:13.248722 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:13.247295 4947 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9733bd5c-945d-4308-92b5-a838e82acf50-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9733bd5c-945d-4308-92b5-a838e82acf50-maas-api-tls\") pod \"9733bd5c-945d-4308-92b5-a838e82acf50\" (UID: \"9733bd5c-945d-4308-92b5-a838e82acf50\") " Nov 03 17:57:13.271723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:13.255143 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9733bd5c-945d-4308-92b5-a838e82acf50-metrics-certs" pod "9733bd5c-945d-4308-92b5-a838e82acf50" (UID: "9733bd5c-945d-4308-92b5-a838e82acf50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:57:13.271723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:13.255465 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9733bd5c-945d-4308-92b5-a838e82acf50-openshift-service-ca" pod "9733bd5c-945d-4308-92b5-a838e82acf50" (UID: "9733bd5c-945d-4308-92b5-a838e82acf50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:57:13.271723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:13.267914 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9733bd5c-945d-4308-92b5-a838e82acf50-kube-api-access-2zrqv" pod "9733bd5c-945d-4308-92b5-a838e82acf50" (UID: "9733bd5c-945d-4308-92b5-a838e82acf50"). InnerVolumeSpecName "kube-api-access-2zrqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:57:13.296135 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:13.295646 4947 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9733bd5c-945d-4308-92b5-a838e82acf50-maas-api-tls" pod "9733bd5c-945d-4308-92b5-a838e82acf50" (UID: "9733bd5c-945d-4308-92b5-a838e82acf50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:57:13.348549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:13.348449 4947 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9733bd5c-945d-4308-92b5-a838e82acf50-metrics-certs\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:57:13.348549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:13.348487 4947 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zrqv\" (UniqueName: \"kubernetes.io/projected/9733bd5c-945d-4308-92b5-a838e82acf50-kube-api-access-2zrqv\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:57:13.348549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:13.348505 4947 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9733bd5c-945d-4308-92b5-a838e82acf50-maas-api-tls\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:57:13.348549 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:13.348519 4947 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9733bd5c-945d-4308-92b5-a838e82acf50-openshift-service-ca\") on node \"ip-10-0-143-64.ec2.internal\" DevicePath \"\"" Nov 03 17:57:13.631770 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:13.631654 4947 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" Nov 03 17:57:13.631770 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:13.631688 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk" event={"ID":"9733bd5c-945d-4308-92b5-a838e82acf50","Type":"ContainerDied","Data":"7f739a25b60f8caebf993097fe6cafb1e1253ccb22b25cd67bcc2338e469ba47"} Nov 03 17:57:13.632413 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:13.631784 4947 scope.go:122] "RemoveContainer" containerID="4f9640d12ee0c8b6b77c2ce4c2b84fa87169cf2a093680330b8358e14a37a7da" Nov 03 17:57:13.687653 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:13.687625 4947 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk"] Nov 03 17:57:13.691889 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:13.691868 4947 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-74d6c75f5f-hwzwk"] Nov 03 17:57:14.564745 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:14.564691 4947 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9733bd5c-945d-4308-92b5-a838e82acf50" path="/var/lib/kubelet/pods/9733bd5c-945d-4308-92b5-a838e82acf50/volumes" Nov 03 17:57:30.129564 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:57:30.129523 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:57:36.677300 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:36.677263 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:57:41.378484 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:41.378446 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:57:44.426823 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:44.426785 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:57:44.427461 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:44.427427 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-9gxvm" podUID="3b61ea26-243b-4e8a-8ca7-3a154a72a9b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:58.365467 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:57:58.365430 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:58:00.129884 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:58:00.129845 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:58:03.094978 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:58:03.094943 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:58:08.355213 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:58:08.355180 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:58:30.128782 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:58:30.128741 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:58:31.194719 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:58:31.194667 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:58:41.475586 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:58:41.475552 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:58:46.065201 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:58:46.065166 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:58:48.909723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:58:48.908813 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-rp689 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:58:48.909723 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:58:48.908884 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-rp689" podUID="43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:51.918887 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:58:51.918847 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:59:00.128589 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:59:00.128548 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:59:09.374605 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:59:09.374571 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:59:17.509436 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:59:17.509404 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:59:17.519383 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:59:17.519361 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 17:59:19.689654 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:59:19.689620 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:59:25.026195 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:59:25.026150 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:59:30.131128 ip-10-0-143-64 kubenswrapper[4947]: E1103 17:59:30.131090 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 17:59:34.059645 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:59:34.059606 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 17:59:52.163200 ip-10-0-143-64 kubenswrapper[4947]: I1103 17:59:52.163167 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 18:00:00.129065 ip-10-0-143-64 kubenswrapper[4947]: E1103 18:00:00.128978 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 18:00:15.792920 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:00:15.792848 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-mjl62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:00:15.805841 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:00:15.805794 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-mjl62" podUID="37849d8a-115b-44a5-bea8-539a92f53653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:00:30.129687 ip-10-0-143-64 kubenswrapper[4947]: E1103 18:00:30.129639 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 18:01:00.130913 ip-10-0-143-64 kubenswrapper[4947]: E1103 18:01:00.130877 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 18:01:17.693723 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:01:17.692797 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6lq95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:01:17.693723 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:01:17.692873 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" podUID="404b1437-2e7e-49c9-a538-23399eeb21eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:01:30.132064 ip-10-0-143-64 kubenswrapper[4947]: E1103 18:01:30.132027 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 18:01:46.085078 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:01:46.085043 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-t6dn2"] Nov 03 18:01:47.695564 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:01:47.695527 4947 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-c9ffffc7f-6lq95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:01:47.696268 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:01:47.696235 4947 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-c9ffffc7f-6lq95" podUID="404b1437-2e7e-49c9-a538-23399eeb21eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:02:00.131921 ip-10-0-143-64 kubenswrapper[4947]: E1103 18:02:00.131883 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 18:02:08.686909 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:08.686874 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7d958cf575-8jkmf_d1ba5407-2f49-4ac7-9b41-157697ab2306/manager/0.log" Nov 03 18:02:09.588422 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:09.588391 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-54fd7b574c-db4h6_61ba1e51-1854-4c3d-9918-d99dd92fbb18/manager/0.log" Nov 03 18:02:10.194378 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:10.194347 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4grck_9f54e03f-0ad5-4368-a9eb-78fad1b70156/proxy/0.log" Nov 03 18:02:11.321186 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:11.321156 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-78d9c46c8c-lm2ft_e17bd3a7-aadb-444c-8835-8e21fdbe1343/maas-api/0.log" Nov 03 18:02:17.914762 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:17.914730 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-t6dn2_879fa899-009b-44d6-adb2-ae2d9a82e427/limitador/0.log" Nov 03 18:02:19.225040 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:19.225010 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zp4kt_d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316/discovery/0.log" Nov 03 18:02:19.411064 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:19.411033 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-645cdd4567-8ghpl_aa392009-346d-4643-8300-74f3258bc223/kube-auth-proxy/0.log" Nov 03 18:02:19.775550 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:19.775516 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-6lq95_404b1437-2e7e-49c9-a538-23399eeb21eb/istio-proxy/0.log" Nov 03 18:02:20.199184 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:20.199114 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-9gxvm_3b61ea26-243b-4e8a-8ca7-3a154a72a9b7/istio-proxy/0.log" Nov 03 18:02:21.024668 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:21.024624 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-mjl62_37849d8a-115b-44a5-bea8-539a92f53653/istio-proxy/0.log" Nov 03 18:02:21.213141 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:21.213105 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-rp689_43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb/istio-proxy/0.log" Nov 03 18:02:22.020541 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:22.020508 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-8544d8cd96-nndq2_ca409853-f894-42a7-b25b-5731b743cf8d/payload-pre-processing/0.log" Nov 03 18:02:22.396887 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:22.396801 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dc56589f5-wsj6w_66b43213-9ec2-4164-9f8d-d7621bd57eb3/router/0.log" Nov 03 18:02:23.380029 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:23.379975 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-b4rmj_3ffcef26-eed6-4a12-86d6-a057a71db539/main/0.log" Nov 03 18:02:23.810033 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:23.810002 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-zxcg7_2eac7cf8-9b38-40e6-8864-09f26a2ff834/main/0.log" Nov 03 18:02:24.182008 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:24.181913 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8hfjq4_6a94473e-1c2c-470e-92a8-1f4ff82791f2/main/0.log" Nov 03 18:02:24.560541 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:24.560515 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-78fhj_6f7be4c6-a940-4bb9-a6f6-076d90494556/main/0.log" Nov 03 18:02:30.129282 ip-10-0-143-64 kubenswrapper[4947]: E1103 18:02:30.129243 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 18:02:34.868834 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:34.868801 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7cwt5/must-gather-wkdbd"] Nov 03 18:02:34.869321 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:34.869304 4947 cpu_manager.go:414] "containerMap: removing container" podUID="9733bd5c-945d-4308-92b5-a838e82acf50" containerName="maas-api" Nov 03 18:02:34.869375 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:34.869323 4947 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9733bd5c-945d-4308-92b5-a838e82acf50" containerName="maas-api" Nov 03 18:02:34.869428 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:34.869383 4947 memory_manager.go:365] "RemoveStaleState removing state" podUID="9733bd5c-945d-4308-92b5-a838e82acf50" containerName="maas-api" Nov 03 18:02:34.872916 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:34.872897 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7cwt5/must-gather-wkdbd" Nov 03 18:02:34.896734 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:34.896688 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7cwt5/must-gather-wkdbd"] Nov 03 18:02:34.904827 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:34.904793 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7cwt5\"/\"default-dockercfg-2ccdh\"" Nov 03 18:02:34.904827 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:34.904793 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7cwt5\"/\"kube-root-ca.crt\"" Nov 03 18:02:34.904827 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:34.904801 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7cwt5\"/\"openshift-service-ca.crt\"" Nov 03 18:02:34.976936 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:34.976908 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpjng\" (UniqueName: \"kubernetes.io/projected/9bda8dcb-0f92-4250-a1fb-0d4a7ef8204e-kube-api-access-jpjng\") pod \"must-gather-wkdbd\" (UID: \"9bda8dcb-0f92-4250-a1fb-0d4a7ef8204e\") " pod="openshift-must-gather-7cwt5/must-gather-wkdbd" Nov 03 18:02:34.977110 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:34.976976 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9bda8dcb-0f92-4250-a1fb-0d4a7ef8204e-must-gather-output\") pod \"must-gather-wkdbd\" (UID: \"9bda8dcb-0f92-4250-a1fb-0d4a7ef8204e\") " pod="openshift-must-gather-7cwt5/must-gather-wkdbd" Nov 03 18:02:35.078148 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:35.078107 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9bda8dcb-0f92-4250-a1fb-0d4a7ef8204e-must-gather-output\") pod \"must-gather-wkdbd\" (UID: \"9bda8dcb-0f92-4250-a1fb-0d4a7ef8204e\") " pod="openshift-must-gather-7cwt5/must-gather-wkdbd" Nov 03 18:02:35.078341 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:35.078230 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpjng\" (UniqueName: \"kubernetes.io/projected/9bda8dcb-0f92-4250-a1fb-0d4a7ef8204e-kube-api-access-jpjng\") pod \"must-gather-wkdbd\" (UID: \"9bda8dcb-0f92-4250-a1fb-0d4a7ef8204e\") " pod="openshift-must-gather-7cwt5/must-gather-wkdbd" Nov 03 18:02:35.078497 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:35.078478 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9bda8dcb-0f92-4250-a1fb-0d4a7ef8204e-must-gather-output\") pod \"must-gather-wkdbd\" (UID: \"9bda8dcb-0f92-4250-a1fb-0d4a7ef8204e\") " pod="openshift-must-gather-7cwt5/must-gather-wkdbd" Nov 03 18:02:35.113581 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:35.113556 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpjng\" (UniqueName: \"kubernetes.io/projected/9bda8dcb-0f92-4250-a1fb-0d4a7ef8204e-kube-api-access-jpjng\") pod \"must-gather-wkdbd\" (UID: \"9bda8dcb-0f92-4250-a1fb-0d4a7ef8204e\") " pod="openshift-must-gather-7cwt5/must-gather-wkdbd" Nov 03 18:02:35.199183 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:35.199069 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7cwt5/must-gather-wkdbd" Nov 03 18:02:35.497620 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:35.497588 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7cwt5/must-gather-wkdbd"] Nov 03 18:02:35.499205 ip-10-0-143-64 kubenswrapper[4947]: W1103 18:02:35.499177 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bda8dcb_0f92_4250_a1fb_0d4a7ef8204e.slice/crio-1d9b365badb9d2dd1d2b9afb0e60dedaa30b81744338a469f1d8472f3f580d7f WatchSource:0}: Error finding container 1d9b365badb9d2dd1d2b9afb0e60dedaa30b81744338a469f1d8472f3f580d7f: Status 404 returned error can't find the container with id 1d9b365badb9d2dd1d2b9afb0e60dedaa30b81744338a469f1d8472f3f580d7f Nov 03 18:02:35.501148 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:35.501125 4947 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:02:35.970278 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:35.970184 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7cwt5/must-gather-wkdbd" event={"ID":"9bda8dcb-0f92-4250-a1fb-0d4a7ef8204e","Type":"ContainerStarted","Data":"1d9b365badb9d2dd1d2b9afb0e60dedaa30b81744338a469f1d8472f3f580d7f"} Nov 03 18:02:39.589688 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:39.589656 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p7vvq_8c96f345-2cda-4554-bef3-be416dc50225/global-pull-secret-syncer/0.log" Nov 03 18:02:39.751157 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:39.751124 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-4j5k5_4cbc83f2-64b1-4804-a180-43df6459ffa4/connection-checker/0.log" Nov 03 18:02:39.857416 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:39.857324 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-hxr8b_4b370319-6de8-41e2-ae5f-21267a2fba74/connection-checker/0.log" Nov 03 18:02:39.972367 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:39.972339 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-j8t6k_78e9f545-8400-48b6-968d-eb65dc94b471/connection-checker/0.log" Nov 03 18:02:40.396024 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:40.395996 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zmzv6_be60fb17-308e-420d-b9e0-d9a05c0c752f/konnectivity-agent/0.log" Nov 03 18:02:40.732295 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:40.732225 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-64.ec2.internal_9d137c53e9ec4edb6f2bb3a754f3bb53/haproxy/0.log" Nov 03 18:02:41.002123 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:41.002088 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7cwt5/must-gather-wkdbd" event={"ID":"9bda8dcb-0f92-4250-a1fb-0d4a7ef8204e","Type":"ContainerStarted","Data":"d30d74c8ee6de1a8f87098113bcae4859002807e7f28c97f474b89431681d081"} Nov 03 18:02:41.002123 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:41.002127 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7cwt5/must-gather-wkdbd" event={"ID":"9bda8dcb-0f92-4250-a1fb-0d4a7ef8204e","Type":"ContainerStarted","Data":"96c8c669f740cfc35eb6a934aee9efa89dd2bc3ac23c851b19a47b73a1905301"} Nov 03 18:02:41.046019 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:41.045962 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-7cwt5/must-gather-wkdbd" podStartSLOduration=1.956409211 podStartE2EDuration="7.045947963s" totalImagesPullingTime="5.089538752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:41.045380141 +0000 UTC m=+6504.634380110" watchObservedRunningTime="2026-11-03 18:02:41.045947963 +0000 UTC m=+6504.634947932" Nov 03 18:02:42.500903 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:42.500871 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7d958cf575-8jkmf_d1ba5407-2f49-4ac7-9b41-157697ab2306/manager/0.log" Nov 03 18:02:43.413511 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:43.413467 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-54fd7b574c-db4h6_61ba1e51-1854-4c3d-9918-d99dd92fbb18/manager/0.log" Nov 03 18:02:43.611310 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:43.611277 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4grck_9f54e03f-0ad5-4368-a9eb-78fad1b70156/proxy/0.log" Nov 03 18:02:45.094576 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:45.094540 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7d958cf575-8jkmf_d1ba5407-2f49-4ac7-9b41-157697ab2306/manager/0.log" Nov 03 18:02:46.060820 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:46.060775 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-54fd7b574c-db4h6_61ba1e51-1854-4c3d-9918-d99dd92fbb18/manager/0.log" Nov 03 18:02:46.306562 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:46.306534 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4grck_9f54e03f-0ad5-4368-a9eb-78fad1b70156/proxy/0.log" Nov 03 18:02:47.711408 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:47.711377 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7d958cf575-8jkmf_d1ba5407-2f49-4ac7-9b41-157697ab2306/manager/0.log" Nov 03 18:02:48.553086 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:48.553056 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-t6dn2_879fa899-009b-44d6-adb2-ae2d9a82e427/limitador/0.log" Nov 03 18:02:48.903318 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:48.903220 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-54fd7b574c-db4h6_61ba1e51-1854-4c3d-9918-d99dd92fbb18/manager/0.log" Nov 03 18:02:49.082411 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:49.082379 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4grck_9f54e03f-0ad5-4368-a9eb-78fad1b70156/proxy/0.log" Nov 03 18:02:50.445962 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.445928 4947 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k"] Nov 03 18:02:50.452496 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.452470 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:50.471676 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.471642 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k"] Nov 03 18:02:50.548908 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.548872 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46df43c2-53b1-4760-b879-6b08e82404f2-sys\") pod \"perf-node-gather-daemonset-vqr7k\" (UID: \"46df43c2-53b1-4760-b879-6b08e82404f2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:50.548908 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.548912 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87hhh\" (UniqueName: \"kubernetes.io/projected/46df43c2-53b1-4760-b879-6b08e82404f2-kube-api-access-87hhh\") pod \"perf-node-gather-daemonset-vqr7k\" (UID: \"46df43c2-53b1-4760-b879-6b08e82404f2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:50.549176 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.549016 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46df43c2-53b1-4760-b879-6b08e82404f2-proc\") pod \"perf-node-gather-daemonset-vqr7k\" (UID: \"46df43c2-53b1-4760-b879-6b08e82404f2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:50.549176 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.549083 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46df43c2-53b1-4760-b879-6b08e82404f2-lib-modules\") pod \"perf-node-gather-daemonset-vqr7k\" (UID: \"46df43c2-53b1-4760-b879-6b08e82404f2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:50.549176 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.549131 4947 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46df43c2-53b1-4760-b879-6b08e82404f2-podres\") pod \"perf-node-gather-daemonset-vqr7k\" (UID: \"46df43c2-53b1-4760-b879-6b08e82404f2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:50.566771 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.566748 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mm7hs\"/\"openshift-service-ca.crt\"" Nov 03 18:02:50.566771 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.566750 4947 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mm7hs\"/\"kube-root-ca.crt\"" Nov 03 18:02:50.567096 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.567080 4947 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mm7hs\"/\"default-dockercfg-z6f5n\"" Nov 03 18:02:50.650472 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.650438 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46df43c2-53b1-4760-b879-6b08e82404f2-proc\") pod \"perf-node-gather-daemonset-vqr7k\" (UID: \"46df43c2-53b1-4760-b879-6b08e82404f2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:50.650667 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.650491 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46df43c2-53b1-4760-b879-6b08e82404f2-lib-modules\") pod \"perf-node-gather-daemonset-vqr7k\" (UID: \"46df43c2-53b1-4760-b879-6b08e82404f2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:50.650667 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.650535 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46df43c2-53b1-4760-b879-6b08e82404f2-podres\") pod \"perf-node-gather-daemonset-vqr7k\" (UID: \"46df43c2-53b1-4760-b879-6b08e82404f2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:50.650667 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.650565 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46df43c2-53b1-4760-b879-6b08e82404f2-proc\") pod \"perf-node-gather-daemonset-vqr7k\" (UID: \"46df43c2-53b1-4760-b879-6b08e82404f2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:50.650667 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.650653 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46df43c2-53b1-4760-b879-6b08e82404f2-lib-modules\") pod \"perf-node-gather-daemonset-vqr7k\" (UID: \"46df43c2-53b1-4760-b879-6b08e82404f2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:50.650954 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.650675 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46df43c2-53b1-4760-b879-6b08e82404f2-sys\") pod \"perf-node-gather-daemonset-vqr7k\" (UID: \"46df43c2-53b1-4760-b879-6b08e82404f2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:50.650954 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.650653 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46df43c2-53b1-4760-b879-6b08e82404f2-podres\") pod \"perf-node-gather-daemonset-vqr7k\" (UID: \"46df43c2-53b1-4760-b879-6b08e82404f2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:50.650954 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.650761 4947 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87hhh\" (UniqueName: \"kubernetes.io/projected/46df43c2-53b1-4760-b879-6b08e82404f2-kube-api-access-87hhh\") pod \"perf-node-gather-daemonset-vqr7k\" (UID: \"46df43c2-53b1-4760-b879-6b08e82404f2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:50.650954 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.650798 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46df43c2-53b1-4760-b879-6b08e82404f2-sys\") pod \"perf-node-gather-daemonset-vqr7k\" (UID: \"46df43c2-53b1-4760-b879-6b08e82404f2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:50.695284 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.695238 4947 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87hhh\" (UniqueName: \"kubernetes.io/projected/46df43c2-53b1-4760-b879-6b08e82404f2-kube-api-access-87hhh\") pod \"perf-node-gather-daemonset-vqr7k\" (UID: \"46df43c2-53b1-4760-b879-6b08e82404f2\") " pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:50.765514 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:50.765482 4947 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:51.075788 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:51.075761 4947 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k"] Nov 03 18:02:51.078605 ip-10-0-143-64 kubenswrapper[4947]: W1103 18:02:51.078553 4947 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod46df43c2_53b1_4760_b879_6b08e82404f2.slice/crio-55e5204a8bf1ea95cc65754f64e022f6fffae494487cbc1b4463b9e77afc9635 WatchSource:0}: Error finding container 55e5204a8bf1ea95cc65754f64e022f6fffae494487cbc1b4463b9e77afc9635: Status 404 returned error can't find the container with id 55e5204a8bf1ea95cc65754f64e022f6fffae494487cbc1b4463b9e77afc9635 Nov 03 18:02:51.111228 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:51.111191 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-bw6bt_e3af86a0-c2cd-4cbb-8143-49e9ae2e9b28/cluster-monitoring-operator/0.log" Nov 03 18:02:51.787457 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:51.787421 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k8vg5_adfb8fe7-01d3-4c9c-a933-df17b1602ca6/node-exporter/0.log" Nov 03 18:02:51.902522 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:51.902497 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k8vg5_adfb8fe7-01d3-4c9c-a933-df17b1602ca6/kube-rbac-proxy/0.log" Nov 03 18:02:52.022468 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:52.022431 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k8vg5_adfb8fe7-01d3-4c9c-a933-df17b1602ca6/init-textfile/0.log" Nov 03 18:02:52.056192 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:52.056100 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" event={"ID":"46df43c2-53b1-4760-b879-6b08e82404f2","Type":"ContainerStarted","Data":"4535c0fa21a81a96e640afd1b443dea9c7469c4407d263513b23f4f42e60677a"} Nov 03 18:02:52.056192 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:52.056183 4947 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" event={"ID":"46df43c2-53b1-4760-b879-6b08e82404f2","Type":"ContainerStarted","Data":"55e5204a8bf1ea95cc65754f64e022f6fffae494487cbc1b4463b9e77afc9635"} Nov 03 18:02:52.056445 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:52.056207 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:52.110341 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:52.110284 4947 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" podStartSLOduration=2.110270064 podStartE2EDuration="2.110270064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:52.109568435 +0000 UTC m=+6515.698568403" watchObservedRunningTime="2026-11-03 18:02:52.110270064 +0000 UTC m=+6515.699270027" Nov 03 18:02:52.440465 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:52.440389 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zp4kt_d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316/discovery/0.log" Nov 03 18:02:52.481201 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:52.481176 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-645cdd4567-8ghpl_aa392009-346d-4643-8300-74f3258bc223/kube-auth-proxy/0.log" Nov 03 18:02:52.611394 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:52.611363 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-6lq95_404b1437-2e7e-49c9-a538-23399eeb21eb/istio-proxy/0.log" Nov 03 18:02:52.727539 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:52.727466 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-9gxvm_3b61ea26-243b-4e8a-8ca7-3a154a72a9b7/istio-proxy/0.log" Nov 03 18:02:52.968783 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:52.968756 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-mjl62_37849d8a-115b-44a5-bea8-539a92f53653/istio-proxy/0.log" Nov 03 18:02:53.028469 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:53.028442 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-rp689_43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb/istio-proxy/0.log" Nov 03 18:02:53.262382 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:53.262356 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-8544d8cd96-nndq2_ca409853-f894-42a7-b25b-5731b743cf8d/payload-pre-processing/0.log" Nov 03 18:02:53.384049 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:53.383978 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dc56589f5-wsj6w_66b43213-9ec2-4164-9f8d-d7621bd57eb3/router/0.log" Nov 03 18:02:56.573453 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:56.573419 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-r755z_2c812633-b15d-4a30-982d-80d276ec78c7/networking-console-plugin/0.log" Nov 03 18:02:57.671017 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:57.670934 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/2.log" Nov 03 18:02:57.676026 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:57.675997 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tnh8w_216c0d66-a3ec-4748-b761-3420536bfa3a/console-operator/3.log" Nov 03 18:02:57.691930 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:57.691905 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-t6dn2_879fa899-009b-44d6-adb2-ae2d9a82e427/limitador/0.log" Nov 03 18:02:58.072021 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:58.071989 4947 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mm7hs/perf-node-gather-daemonset-vqr7k" Nov 03 18:02:59.768539 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:02:59.768508 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-88dnt_fa01a744-9121-4f6a-b280-dba215c1ab8f/volume-data-source-validator/0.log" Nov 03 18:03:00.129106 ip-10-0-143-64 kubenswrapper[4947]: E1103 18:03:00.129018 4947 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rj5rd" containerName="extractor" Nov 03 18:03:02.384918 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:03:02.384885 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xfk72_5ced7a93-d9f0-44de-9a76-34a84e41843b/dns/0.log" Nov 03 18:03:02.482100 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:03:02.482065 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xfk72_5ced7a93-d9f0-44de-9a76-34a84e41843b/kube-rbac-proxy/0.log" Nov 03 18:03:02.849499 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:03:02.849393 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zfshj_3cbd4c91-69f0-47be-94be-e451d44b4992/dns-node-resolver/0.log" Nov 03 18:03:04.455124 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:03:04.455081 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bflh2_812067ca-eef5-4ab1-a24e-61796ac0721b/node-ca/0.log" Nov 03 18:03:05.564411 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:03:05.564376 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zp4kt_d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316/discovery/0.log" Nov 03 18:03:05.603795 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:03:05.603760 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-645cdd4567-8ghpl_aa392009-346d-4643-8300-74f3258bc223/kube-auth-proxy/0.log" Nov 03 18:03:05.734860 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:03:05.734833 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-6lq95_404b1437-2e7e-49c9-a538-23399eeb21eb/istio-proxy/0.log" Nov 03 18:03:05.879560 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:03:05.879454 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-9gxvm_3b61ea26-243b-4e8a-8ca7-3a154a72a9b7/istio-proxy/0.log" Nov 03 18:03:06.179313 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:03:06.179207 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-mjl62_37849d8a-115b-44a5-bea8-539a92f53653/istio-proxy/0.log" Nov 03 18:03:06.248143 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:03:06.248106 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-rp689_43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb/istio-proxy/0.log" Nov 03 18:03:06.566348 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:03:06.566311 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-8544d8cd96-nndq2_ca409853-f894-42a7-b25b-5731b743cf8d/payload-pre-processing/0.log" Nov 03 18:03:06.714248 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:03:06.714216 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dc56589f5-wsj6w_66b43213-9ec2-4164-9f8d-d7621bd57eb3/router/0.log" Nov 03 18:03:07.842936 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:03:07.842896 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zp4kt_d7e6638a-7e4f-4ea6-aa89-2d6a0ec49316/discovery/0.log" Nov 03 18:03:08.427633 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:03:08.427602 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-645cdd4567-8ghpl_aa392009-346d-4643-8300-74f3258bc223/kube-auth-proxy/0.log" Nov 03 18:03:08.641252 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:03:08.641220 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-6lq95_404b1437-2e7e-49c9-a538-23399eeb21eb/istio-proxy/0.log" Nov 03 18:03:08.861500 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:03:08.861469 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-9gxvm_3b61ea26-243b-4e8a-8ca7-3a154a72a9b7/istio-proxy/0.log" Nov 03 18:03:09.320006 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:03:09.319965 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-mjl62_37849d8a-115b-44a5-bea8-539a92f53653/istio-proxy/0.log" Nov 03 18:03:09.437991 ip-10-0-143-64 kubenswrapper[4947]: I1103 18:03:09.437962 4947 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-c9ffffc7f-rp689_43109fe3-3fd2-4cb2-be1e-b9d5e2e541fb/istio-proxy/0.log"